Objective

2

Depth map plays an increasingly important role in many computer vision applications

such as 3D reconstruction

augmented reality

and gesture recognition.A new generation of active 3D range sensors

such as Microsoft Kinect camera

enables the acquisition of a real-time and affordable depth map.Incidentally

unlike natural images captured by RGB sensors

the depth maps captured by range sensors typically have low resolution (LR) and inaccurate edges due to their intrinsic physical constraints.Given that an accurate and high-resolution (HR) depth map is required and preferable in many applications

excellent depth map super-resolution (SR) techniques are desirable.Depth map SR can be generally addressed by two different types of approaches that depend on the use of input data.For single depth map SR

the resolution of the input depth map can be enhanced based on the information learned with from a pre-collected training database.Meanwhile

depth map SR algorithms that use RGB-D data can be further classified into MRF and filtering-based approaches.MRF-based methods view depth map SR as an optimization problem.Filtering-based methods obtain the weighted average of local depth map pixels for SR purposes.These methods aim to obtain a smooth HR depth map for regions belonging to the same object.However

these methods have two main issues:1) the inaccurate edges of the depth map cannot be fully refined and 2) the edges of the HR depth map suffer from blurring.In this paper

a novel texture edge-guided depth reconstruction approach is proposed to address the issue of existing methods.We pay more attention to the depth edge refinement

which is usually ignored by existing methods.

Method

2

In the first stage

an initial HR depth map is obtained by general up-sampling methods

such as interpolation and filters.Then

initial depth edges are extracted from the initial HR depth map by using many edge detectors for edge detection

such as Sobel and Canny.The edges extracted directly from the initial HR depth map are not the true edges because the misalignment between the LR depth map edges and the texture edges and the up-sampling operation can cause further edge errors.Subsequently

the texture edges are extracted from the color image.Traditional approaches for edge detection do not consider the visually salient edges; the texture edges and illusory contours are all taken as image edges.Moreover

many edges of the color image do not correspond to depth edges

such as the edges inside the object.Inspired by the advanced positive result of the vision field

we propose a depth map edge detection method based on the structured forest.The edge map of the color image is initially extracted by using the recently structured learning approach.By incorporating the 3D space information provided by the initial HR depth map

the texture edges of the objects inside are removed.Then

we obtain a clear and true depth edge map.Finally

the depth values on each side of the depth edge are refined to align the depth edges and correct the depth errors in the initial HR depth map.We detect the incorrect depth regions between the initial depth edges and the corresponding true depth edges and then fill the incorrect regions until the depth edges are consistent with the corresponding color image.The incorrect regions of initial HR depth map are refined by the joint bilateral filter in an outside-inward refining order that is regularized by the detected true depth edges.

Result

2

We perform experiments on the NYU dataset

which offers real-world color-depth image pairs that were captured by a Kinect camera.To evaluate the performance of our proposed method

we compare our results with two method categories:1) state-of-the-art single depth image super resolution methods (ScSR

PB

and E.G.) and 2) state-of-the-art color-guided depth map super resolution approaches (JBU

GIU

MRF

WMF

and JTU).We implement most of these methods by using the same parameter settings as provided in the corresponding papers.We down-sample the original depth maps into LR ones and perform SR.We evaluate our proposed method with the recovered HR depth map and the reconstructed point clouds.The recovered HR depth maps indicate that our proposed methods generate more visually appealing results than the compared approaches.The boundaries in our results are generally sharper and smoother along the edge direction

whereas the compared methods suffer from blurred artifacts around the boundaries.To demonstrate further the effectiveness of our proposed approach

we provide the 3D point clouds constructed from the up-scaled depth map with different methods.Results indicate that our proposed method yields a relatively clear foreground and background

while the competing results suffer from obvious flying pixels and aliased planes.

Conclusion

2

We present a novel method for depth map SR for Kinect depth.Experimental results demonstrate that the proposed method provides sharp and clear edges for the Kinect depth

and the depth edges are aligned with the texture edges.The proposed framework synthesizes an HR depth map given its LR depth map and corresponding HR color image.Our proposed method first estimates the initial HR depth map via traditional up-sampling approaches

then extracts the true edges of the RGB-D data and the fake edges of the initial HR depth map to identify the incorrect regions between the two edges.The incorrect regions of the initial HR depth maps are further refined by joint bilateral filter in an outside-inward refining order to align the edges of color image and depth map.The key to our success is the use of RGB-D depth edge detection

which is inspired by the structured forests-based edge detection.Besides

unlike most depth enhancement methods that use raster-scan order to fill incorrect regions

our method can determine the filing order by considering the true edges.Thus

our HR depth map output exhibits better quality with clear and aligned depth edges compared with the existing depth map SR.However

texture-based guidance may result in incorrect depth value due to the smooth object surface with rich color texture.Thus

the suppression of texture copying artifacts may be our next research goal.