Current Issue Cover
纹理边缘引导的深度图像超分辨率重建

李宇翔,邓慧萍(武汉科技大学信息科学与工程学院)

摘 要
目的:深度图像作为一种普遍的三维场景信息表达方式在立体视觉领域有着广泛的应用。Kinect深度相机能够实时获取场景的深度图像,但由于内部硬件的限制和外界因素的干扰,获取的深度图像存在分辨率低、边缘不准确的问题,无法满足实际应用的需要。为此提出了一种基于彩色图像边缘引导的Kinect深度图像超分辨率重建算法。方法:首先对深度图像进行初始化上采样,并提取初始化深度图像的边缘;进一步利用高分辨率彩色图像和深度图像的相似性,采用基于结构化学习的边缘检测方法提取深度图的正确边缘;最后找出初始化深度图的错误边缘和深度图正确边缘之间的不可靠区域,采用边缘对齐的策略对不可靠区域进行插值填充。结果:在NYU2数据集上进行实验,与8种最新的深度图像超分辨率重建算法作比较,用重建之后的深度图像和三维重建的点云效果进行验证。实验结果表明本文算法在提高深度图像的分辨率的同时,能有效修正上采样后深度图像的边缘,使深度边缘与纹理边缘对齐,也能抑制上采样算法带来的边缘模糊现象;三维点云效果显示,本文算法能准确区分场景中的前景和背景,应用于三维重建等应用能取得较其他算法更好的效果。结论:本文算法普遍适用于Kinect深度图像的超分辨率重建问题,该算法结合同场景彩色图像与深度图像的相似性,利用纹理边缘引导深度图像的超分辨率重建,可以得到较好的重建结果。
关键词
Depth map super-resolution reconstruction based on the texture edge guided

li yu xiang,denghuiping(Wuhan University of Science and Technology)

Abstract
Objective Depth map is playing a more and more important role in many computer vision applications such as 3D reconstruction, augmented reality, gesture recognition, etc. A new generation of active 3D range sensors, such as Microsoft Kinect camera enables getting depth map with real time and affordable cost. Unfortunately, compared to the natural images captured by the RGB sensor, depth maps captured by the range sensor are typically with low resolution (LR) and inaccurate edge due to intrinsic physical constraints. Since accurate depth map at high resolution (HR) is required and preferable in many applications, the techniques of excellent depth map super-resolution (SR) would be desirable. Depth map SR can be generally addressed by two different types of approaches, depending on the use of input data. For single depth map SR, the resolution of the input depth map can be enhanced based on the information learned with from a pre-collected training database. On the other hand, depth map SR algorithm using RGB-D data can be further classified into MRF and filtering-based approaches. MRF-based methods viewed depth map SR as an optimization problem. Filtering-based methods perform weighted average of local depth map pixels for SR purposes. These methods aimed at obtaining a smooth HR depth map for regions belonging to the same object. However, two main issues of these methods are: 1) inaccurate edges in depth map can’t be well refined; 2) the HR depth map still suffer from blurring edges. In this paper, a novel texture edge guided depth reconstructing approach is proposed to tackle the issue of existing methods. We pay more attention to the depth edge refinement, which is usually ignored by existing methods. Method In the first stage, an initial HR depth map is obtained by general up-sampling methods such as interpolation and filters. Besides, initial depth edges are extracted from the initial HR depth map, many edge detectors can be used for edge detection, such as Sobel and Canny. Since the misalignment between LR depth map edges and texture edges, and up-sampling operation can cause further error of edge, the edges extracted from the initial HR depth map directly are not the true edges. In the next step, texture edges are extracted from the color image. Traditional approaches for edge detection don’t consider visually salient edges, the texture edges and illusory contours are all taken as image edges. Moreover, many edges of color image do not correspond to depth edges, such as edges inside object. Inspired by the advanced positive result of vision field, we propose a depth map edge detection method based on structured forest. Edge map of color image is extracted firstly by using the recently structured learning approach. By incorporating 3-D space information provided by the initial HR depth map, texture edges existing in objects’ inside are removed. Then, we get a clear and true depth edge map. Finally, depth values on each side of the depth edge are refined to align the depth edges and correct the depth errors in the initial HR depth map. We detect the incorrect depth regions between initial depth edges and corresponding true depth edges, and then fill the incorrect regions until the depth edges are consistent with the corresponding color image. The incorrect regions of initial HR depth map is refined by the joint bilateral filter in an outside-inward refining order regularized by detected true depth edges. Result We perform experiments on NYU dataset which offers real-world color-depth image pairs that are captured by Kinect camera. To evaluate the performance of our proposed method, we compare our results with two categories of the methods. 1) State-of-the-art single depth image super resolution methods: ScSR, PB, EG. 2) State-of-the-art color guided depth map super resolution approaches: JBU, GIU, MRF, WMF, JTU. We implemented most of these methods by using the same parameter settings as provided in the corresponding papers. We down-sample the original depth maps into LR ones and performing SR. We evaluate our proposed method with both recovered HR depth map and reconstructed point clouds. The recovered HR depth maps indicate that our proposed methods generate more visual appealing results than the compared approaches. Boundaries in our results are generally sharper and smoother along the edge direction. While the compared methods suffer from blur artifacts around boundaries visually. To further demonstrate the effectiveness of our proposed approach, we provide the 3D point clouds constructed from the up-scaled depth map with different methods. The results indicate that our proposed method yields relatively clear foreground and background, while the completing results suffer from obvious flying pixels and aliased planes. Conclusion We presented a novel method for depth map SR for Kinect depth. Experimental results demonstrate that the proposed method provides sharp and clear edges for the Kinect depth, and depth edges are aligned with the texture edges. The proposed framework synthesizes an HR depth map given its LR depth map and corresponding HR color image. Our proposed method first estimates an initial HR depth map via traditional upsampling approaches, then extract the true edges of RGB-D data and the fake edges of the initial HR depth map to identify the incorrect regions between the two edges. The incorrect regions of initial HR depth maps are further refined by joint bilateral filter in an outside-inward refining order so that the edges of color image and depth map are alignment. Key to our success is the use of the RGB-D depth edge detection, inspired by the structured forests-based edge detection. Besides, different from most depth enhancement methods that using raster-scan order to fill incorrect regions, our method is able to determine the filing order by taking the true edges into consideration. This is the reason why, compared to existing depth map SR, our produced HR depth map output exhibits improved quality with clear and aligned depth edges. However, texture based guidance might result in incorrect depth value due to a smooth object surface with rich color texture, and thus how to suppress texture copying artifacts is expected as our next research goal.
Keywords
QQ在线


订阅号|日报