浏览全部资源
扫码关注微信
收稿：2018-11-07，
修回：2019-1-22，
纸质出版：2019-07-16
移动端阅览
目的
2
深度相机能够对场景的深度信息进行实时动态捕捉，但捕获的深度图像分辨率低且容易形成空洞。利用高分辨率彩色图像作为引导，是深度图超分辨率重建的重要方式。现有方法对彩色边缘与深度不连续区域的不一致性问题难以有效解决，在深度图超分辨率重建中引入了纹理复制伪影。针对这一问题，本文提出了一种鲁棒的彩色图像引导的深度图超分辨率重建算法。
方法
2
首先，利用彩色图像边缘与深度图像边缘的结构相关性，提出RGB-D结构相似性度量，检测彩色图像与深度图像共有的边缘不连续区域，并利用RGB-D结构相似性度量自适应选取估计像素点邻域的最优图像块。接着，通过提出的定向非局部均值权重，在图像块区域内建立多边引导下的深度估计，解决彩色边缘和深度不连续区域的结构不一致性。最后，利用RGB-D结构相似性度量与图像平滑性之间的对应关系，对多边引导权重的参数进行自适应调节，实现鲁棒的深度图超分辨率重建。
结果
2
在Middlebury合成数据集、ToF和Kinect数据集以及本文自建数据集上的实验结果表明，相比其他先进方法，本文方法能够有效抑制纹理复制伪影。在Middlebury、ToF和Kinect数据集上，本文方法相较于次优算法，平均绝对偏差平均降低约63.51%、39.47 %和7.04 %。
结论
2
对于合成数据集以及真实场景的深度数据集，本文方法均能有效处理存在于彩色边缘和深度不连续区域的不一致性问题，更好地保留深度边缘的不连续性。
Objective
2
Depth cameras can capture depth images of the dynamic scene in real time
which exhibits unique advantages in depth information acquisition. However
the depth images are often sensitive to noise and subject to low spatial resolution
and depth values are missing in some areas. Depth information and color edges are two complementary aspects that describe the same scene and therefore have a strong correlation in structure. Depth discontinuous transitions often coincide with color transitions. The color-depth correlation can be used for depth image reconstruction due to the blurred edges of low resolution depth images. Utilizing a high resolution color image as a reference is an important approach for reconstructing a high-resolution depth image. However
rich texture regions can be found in color images
except depth images. Among the highly challenging problems in color-guided depth image reconstruction is the inconsistency between color edges and depth discontinuities in the texture region. Simply passing structural information in the color image to the target one could introduce significant errors. Existing methods tend to consider only color images and ignore the correlation with depth images
which ineffectively solves the inconsistency
resulting in texture copying artifacts and even blurred depth edges. In this paper
we propose a color-guided depth image super-resolution reconstruction algorithm that is robust to inconsistency.
Method
2
We propose RGB-D structure similarity measure to predict the color edges that are most likely to coincide with depth discontinuities by using the structural correlation between the color and depth images. We examine local structural gradients rather than the gradient magnitude of individual pixel to measure the inconsistency effectively. Result shows that the proposed RGB-D structure similarity measure is less affected by color texture. We use the proposed RGD-D structure similarity measure as an image patch adaptive selection indicator
which can effectively reflect the discontinuity of the depth edges. The conventional image patch is centered on the pixel to be estimated. However
when the pixel is located in the depth edge region
the depth estimation is blurred due to the changes in the nearby depth gradients. In contrast with the conventional image patches
we select the optimal image patch that is least likely to contain prominent depth edges among all the image patches in the pixel neighborhood
which helps preserve sharp depth edges. Then
the multilateral guided estimation of depth values is performed in the selected optimal patch. We propose an oriented nonlocal means weighting scheme using high-quality structural gradients and directional information by utilizing the nonlocal characteristics of the color and the depth images. The weighting scheme combines with spatial and range kernel as the multilateral guidance for depth estimation
which effectively solves the structural inconsistency
preserves depth discontinuities
and is robust to depth holes. Finally
the three bandwidth parameters of our multilateral guidance weighting scheme are important benchmarks of our depth image super-resolution reconstruction model. The proposed RGB-D structure similarity measure is related to the depth image smoothness
which corresponds to depth discontinuity and is less affected by the incoherent texture. Small bandwidth parameters can effectively preserve depth discontinuities but poorly perform in smoothing noise. Large bandwidth parameters can effectively smooth noise but may blur depth discontinuities. We can adaptively adjust the multilateral guidance weight parameters in accordance with the RGB-D structure similarity measure to achieve robust depth image construction. The framework of our depth image super-resolution reconstruction is based on the multilateral guidance. Moreover
the corresponding relationship between the proposed RGB-D structure similarity measure and image smoothness is used to select the position of the neighborhood image patch and the size of the guide weight parameters adaptively.
Result
2
Quantitative and qualitative evaluation results show that our method has promising performance compared with other state-of-the-art methods on Middlebury synthetic
ToF real
and Kinect real datasets and our own dataset. Our method can effectively suppress texture copying artifacts
restore the depth hole image
and preserve depth discontinuities. We use the mean absolute difference as an evaluation metric
which is also the commonly used evaluation analysis metric for the depth image reconstruction. In particular
the mean absolute difference of the proposed method is decreased by approximately 63.51%
39.47%
and 7.04% on the Middlebury
ToF
and Kinect datasets on average compared with the suboptimal algorithm. Furthermore
when the up-sampling factor of depth image reconstruction increases
the results of our reconstruction are more evident than those of other state-of-the-art methods as we fully utilize the structural information of color images. The other methods ineffectively solve the influence of color textures
whereas the depth image information is no longer reliable. For the depth hole image
most of the previous methods can only restore the depth image but cannot increase the depth image resolution
or the two can be separated. Our method can effectively restore the depth hole image
and the depth image super-resolution reconstruction and experiments on the NYU raw dataset verify the effectiveness of our method.
Conclusion
2
Our method can effectively handle the inconsistency between color edges and depth discontinuities in color-guided depth image super-resolution reconstruction
effectively restoring depth holes. In particular
our method can be used for not only synthetic datasets but also real-world depth datasets to preserve depth discontinuities effectively.
Nakagawa Y, Kihara K, Tadoh R, et al. Super resolving of the depth map for 3D reconstruction of underwater terrain using kinect[C]//Proceedings of the 22nd International Conference on Parallel and Distributed Systems. Wuhan, China: IEEE, 2016: 1237-1240.[ DOI:10.1109/icpads.2016.0168 http://dx.doi.org/10.1109/icpads.2016.0168 ]
Häne C, Zach C, Lim J, et al. Stereo depth map fusion for robot navigation[C]//Proceedings of 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. San Francisco, CA: IEEE, 2011: 1618-1625.[ DOI:10.1109/iros.2011.6094704 http://dx.doi.org/10.1109/iros.2011.6094704 ]
Chen C F, Bolas M, Rosenberg E S. Rapid creation of photorealistic virtual reality content with consumer depth cameras[C]//Proceedings of 2017 IEEE Virtual Reality. Los Angeles, CA: IEEE, 2017: 473-474.[ DOI:10.1109/vr.2017.7892385 http://dx.doi.org/10.1109/vr.2017.7892385 ]
Yang Y X, Gao M Y, Yin K, et al. High-qualitydepth map reconstruction combining stereo image pair[J]. Journal of Image and Graphics, 2015, 20(1):1-10.
杨宇翔, 高明煜, 尹克, 等.结合同场景立体图对的高质量深度图像重建[J].中国图象图形学报, 2015, 20(1):1-10. [DOI:10.11834/jig.20150101]
Xiao L, Heide F, O'Toole M, et al. Defocus deblurring and superresolution for time-of-flight depth cameras[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE, 2015: 2376-2384.[ DOI:10.1109/cvpr.2015.7298851 http://dx.doi.org/10.1109/cvpr.2015.7298851 ]
Smisek J, Jancosek M, Pajdla T. 3D with Kinect[M]//Fossati A, Gall J, Grabner H, et al. Consumer Depth Cameras for Computer Vision: Research Topics and Applications. London: Springer, 2013: 3-25.[ DOI:10.1007/978-1-4471-4640-7_1 http://dx.doi.org/10.1007/978-1-4471-4640-7_1 ]
Guðmundsson S Á, Aanaes H, Larsen R. Fusion of stereo vision and time-of-flight imaging for improved 3d estimation[J]. International Journal of Intelligent Systems Technologies and Applications, 2008, 5(3-4):425-433.[DOI:10.1504/ijista.2008.021305]
Li L. Time-of-flight camera——an introduction[R]. Technical White Paper SLOA190B. Dallas: Texas Texas Instruments, 2014.
Hansard M, Lee S, Choi O, et al. Time-of-Flight Cameras:Principles, Methods and Applications[M]. London:Springer, 2013.[DOI:10.1007/978-1-4471-4658-2]
Gu S H, Zuo W M, Guo S, et al. Learning dynamic guidance for depth image enhancement[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 712-721.[ DOI:10.1109/cvpr.2017.83 http://dx.doi.org/10.1109/cvpr.2017.83 ]
Li Y J, Huang J B, Ahuja N, et al. Deep joint image filtering[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer, 2016: 154-169.[ DOI:10.1007/978-3-319-46493-0_10 http://dx.doi.org/10.1007/978-3-319-46493-0_10 ]
Bartczak B, Koch R. Dense depth maps from low resolution time-of-flight depth and high resolution color views[C]//The 5th International Symposium on Advances in Visual Computing. Las Vegas, NV: Springer, 2009: 228-239.[ DOI:10.1007/978-3-642-10520-3_21 http://dx.doi.org/10.1007/978-3-642-10520-3_21 ]
Kopf J, Cohen M F, Lischinski D, et al. Joint bilateral upsampling[J]. ACM Transactions on Graphics, 2007, 26(3):#96.[DOI:10.1145/1276377.1276497]
Shen X Y, Zhou C, Xu L, et al. Mutual-structure for joint filtering[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 3406-3414.[ DOI:10.1109/iccv.2015.389 http://dx.doi.org/10.1109/iccv.2015.389 ]
He K M, Sun J, Tang X O. Guided image filtering[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6):1397-1409.[DOI:10.1109/TPAMI.2012.213]
Ferstl D, Reinbacher C, Ranftl R, et al. Image guided depth upsampling using anisotropic total generalized variation[C]//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, NSW: IEEE, 2013: 993-1000.[ DOI:10.1109/iccv.2013.127 http://dx.doi.org/10.1109/iccv.2013.127 ]
Diebel J, Thrun S. An application of Markov random fields to range sensing[C]//Proceedings of the 18th International Conference on Neural Information Processing Systems. Cambridge, MA: MIT Press, 2005: 291-298.
Harrison A, Newman P. Image and sparse laser fusion for dense scene reconstruction[C]//Proceedings of the 7th International Conference on Field and Service Robotics. Berlin, Heidelberg: Springer, 2010: 219-228.[ DOI:10.1007/978-3-642-13408-1_20 http://dx.doi.org/10.1007/978-3-642-13408-1_20 ]
Park J, Kim H, Tai Y W, et al. High quality depth map upsampling for 3D-TOF cameras[C]//Proceedings of 2011 International Conference on Computer Vision. Barcelona, Spain: IEEE, 2011: 1623-1630.[ DOI:10.1109/iccv.2011.6126423 http://dx.doi.org/10.1109/iccv.2011.6126423 ]
Yang J Y, Ye X C, Li K, et al. Depth recovery using an adaptive color-guided auto-regressive model[C]//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy: Springer, 2012: 158-171.[ DOI:10.1007/978-3-642-33715-4_12 http://dx.doi.org/10.1007/978-3-642-33715-4_12 ]
Chan D, Buisman H, Theobalt C, et al. A noise-aware filter for real-time depth upsampling[C]//Proceedings of Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications. Marseille, France: HAL, 2008.
Yang Q X, Yang R G, Davis J, et al. Spatial-depth super resolution for range images[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN: IEEE, 2007: 1-8.[ DOI:10.1109/cvpr.2007.383211 http://dx.doi.org/10.1109/cvpr.2007.383211 ]
Fukushima N, Takeuchi K, Kojima A. Self-similarity matching with predictive linear upsampling for depth map[C]//Proceedings of 20163DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video. Hamburg, Germany: IEEE, 2016: 1-4.[ DOI:10.1109/3dtv.2016.7548889 http://dx.doi.org/10.1109/3dtv.2016.7548889 ]
Min D B, Lu J B, Do M N. Depth video enhancement based on weighted mode filtering[J]. IEEE Transactions on Image Processing, 2012, 21(3):1176-1190.[DOI:10.1109/tip.2011.2163164]
Zuo X X, Zheng J B. A refined weighted mode filtering approach for depth video enhancement[C]//Proceedings of 2013 International Conference on Virtual Reality and Visualization. Xi'an, China: IEEE, 2013: 138-144.[ DOI:10.1109/icvrv.2013.30 http://dx.doi.org/10.1109/icvrv.2013.30 ]
Li Z G, Zheng J H, Zhu Z J, et al. Weighted guided image filtering[J]. IEEE Transactions on Image Processing, 2015, 24(1):120-129.[DOI:10.1109/TIP.2014.2371234]
Chen J, Tang C K, Wang J. Noise brush:interactive high quality image-noise separation[J]. ACM Transactions on Graphics, 2009, 28(5):#146.[DOI:10.1145/1661412.1618492]
Scharstein D, Pal C. Learning conditional random fields for stereo[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN: IEEE, 2007: 1-8.[ DOI:10.1109/cvpr.2007.383191 http://dx.doi.org/10.1109/cvpr.2007.383191 ]
Hirschmuller H, Scharstein D. Evaluation of cost functions for stereo matching[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN: IEEE, 2007: 1-8.[ DOI:10.1109/cvpr.2007.383248 http://dx.doi.org/10.1109/cvpr.2007.383248 ]
Silberman N, Hoiem D, Kohli P, et al. Indoor segmentation and support inference from RGBD images[C]//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy: Springer, 2012: 746-760.[ DOI:10.1007/978-3-642-33715-4_54 http://dx.doi.org/10.1007/978-3-642-33715-4_54 ]
Liu M Y, Tuzel O, Taguchi Y. Joint geodesic upsampling of depth images[C]//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR: IEEE, 2013: 169-176.[ DOI:10.1109/cvpr.2013.29 http://dx.doi.org/10.1109/cvpr.2013.29 ]
Liu W, Chen X G, Yang J, et al. Robust color guided depth map restoration[J]. IEEE Transactions on Image Processing, 2017, 26(1):315-327.[DOI:10.1109/tip.2016.2612826]
Ham B, Cho M, Ponce J. Robust image filtering using joint static and dynamic guidance[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE, 2015: 4823-4831.[ DOI:10.1109/cvpr.2015.7299115 http://dx.doi.org/10.1109/cvpr.2015.7299115 ]
相关作者
相关机构
京公网安备11010802024621
