Depth image recovery based on dual-scale sequential optimized filling

Dongyue Chen; Xiaoming Zhu; Teng Ma; Yuanyuan Song; Tong Jia

doi:10.11834/jig.210048

Image Processing and Coding | Views : 0 下载量: 0 CSCD: 0

PDF
Export
Share
Collection
Album

Depth image recovery based on dual-scale sequential optimized filling
Vol. 27, Issue 8, Pages: 2344-2355(2022)
Published： 16 August 2022 ，

Accepted： 19 May 2021
DOI： 10.11834/jig.210048
稿件说明：

移动端阅览

Dongyue Chen, Xiaoming Zhu, Teng Ma, Yuanyuan Song, Tong Jia. Depth image recovery based on dual-scale sequential optimized filling. [J]. Journal of Image and Graphics 27(8):2344-2355(2022)
DOI：

Dongyue Chen, Xiaoming Zhu, Teng Ma, Yuanyuan Song, Tong Jia. Depth image recovery based on dual-scale sequential optimized filling. [J]. Journal of Image and Graphics 27(8):2344-2355(2022) DOI： 10.11834/jig.210048.

摘要

目的

深度图像作为一种重要的视觉感知数据，其质量对于3维视觉系统至关重要。由于传统方法获取的深度图像大多有使用场景的限制，容易受到噪声和环境影响，导致深度图像缺失部分深度信息，使得修复深度图像仍然是一个值得研究并有待解决的问题。对此，本文提出一种用于深度图像修复的双尺度顺序填充框架。

方法

首先，提出基于条件熵快速逼近的填充优先级估计算法。其次，采用最大似然估计实现缺失深度值的最优预测。最后，在像素和超像素两个尺度上对修复结果进行整合，准确实现了深度图像孔洞填充。

结果

本文方法在主流数据集MB(Middlebury)上与7种方法进行比较，平均峰值信噪比(peak signal-to-noise ratio，PSNR)和平均结构相似性指数(structural similarity index，SSIM)分别为47.955 dB和0.998 2；在手工填充的数据集MB+中，本文方法的PSNR平均值为34.697 dB，SSIM平均值为0.978 5，对比其他算法，本文深度修复效果有较大优势。在时间效率对比实验中，本文方法也表现优异，具有较高的效率。在消融实验部分，对本文提出的填充优先级估计、深度值预测和双尺度改进分别进行评估，验证了本文创新点的有效性。

结论

实验结果表明，本文方法在鲁棒性、精确度和效率方面相较于现有方法具有比较明显的优势。

Abstract

Objective

The acquired depth information has led to the research development of three-dimensional reconstruction and stereo vision. However

the acquired depth images issues have challenged of image holes and image noise due to the lack of depth information. The quality of the depth image is as a benched data source for each 3D-vision(3DV) system. Our method is focused on the lack of depth map information repair derived from objective factors in the depth acquisition process. It is required of the high precision

the spatial distribution difference between color and depth features

the interference of noise and blur

and the large scale holes information loss.

Method

Real-time ability is relatively crucial in terms of the depth image recovery algorithms serving as pre-processing modules in the 3DV systems. The sequential filling method has been optimized in computational speed by processing each invalid point in one loop. The invalid points based pixels are obtained without depth values. By contrast

depth values captured pixels are referred to as valid points. Therefore

we facilitate a dual-scale sequential filling framework for depth image recovery. We carry out filling priority estimation and depth value prediction of the invalid points in this framework. For the evaluation of the priority of invalid points

we use conditional entropy as the benchmark for evaluating the priority of invalid point filling evaluation and verification. It is incredible to estimate the filling priority and filling depth value through the overall features of a single pixel and its 8-neighborhood. However

the use of multi-scale filtering increases the computational costs severely. We introduce the super-pixel over-segmentation algorithm to segment the input image into more small patches

which ensures the pixels inside the super-pixel homogeneous contexts like color

texture

and depth. We believe that the super-pixels can provide more reliable features in larger scale for priority estimation filling and depth value prediction. In addition

we optioned a simple linear iterative clustering (SLIC) algorithm to handle the super-pixel segmentation task and added a depth difference metric for the image characteristics of RGB-D to make it efficient and reliable. For depth estimation

we use maximum likelihood estimation to estimate the depth of invalid points integrated to the depth value exhaustive method. Finally

the restoration results are integrated on the pixel and super-pixel scales to accurately fill the holes in the depth image.

Result

Our method is compared to 7 methods related to dataset Middlebury (MB)

which shows great advantages on deep repair effection. The averaged peak signal-to-noise ratio (PSNR) is 47.955 dB and the averaged structural similarity index (SSIM) is 0.998 2. Our PSNR reached 34.697 dB and the SSIM reached 0.978 5 in MB based manual populated data set for deep repair. The method herein verifies that this algorithm has relatively strong efficiency in comparison to time efficiency validation. Our filling priority estimation

depth value prediction and double-scale improvement ability are evaluated in the ablation experimental section separately.

Conclusion

We illustrate a dual-scale sequential filling framework for depth image recovery. The experimental results demonstrate that our algorithm proposed has its priority to optimize robustness

precision and efficiency.

关键词

深度图像修复顺序填充条件熵快速逼近深度最优预测超像素

Keywords

depth image recoverysequential fillingfast approximation of conditional entropydepth value predictionsuper-pixel

references

Achanta R, Shaji A, Smith K, Lucchi A, Fua P and Süsstrunk S. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11): 2274-2282 [DOI: 10.1109/tpami.2012.120]

Gong X J, Liu J Y, Zhou W H and Liu J L. 2013. Guided depth enhancement via a fast marching method.Image and Vision Computing, 31(10): 695-703 [DOI: 10.1016/j.imavis.2013.07.006]

Harrison A and Newman P. 2010. Image and sparse laser fusion for dense scene reconstruction//Proceedings of Field and Service Robotics. Berlin, Germany: Springer: 219-228 [DOI: 10.1007/978-3-642-13408-1_20http://dx.doi.org/10.1007/978-3-642-13408-1_20]

Hershey J R and Olsen P A. 2007. Approximating the Kullback Leibler divergence between Gaussian mixture models//Proceedings of 2007 IEEE International Conference on Acoustics, Speech and Signal Processing—ICASSP'07. Honolulu, USA: IEEE: IV-317-IV-320 [DOI: 10.1109/icassp.2007.366913http://dx.doi.org/10.1109/icassp.2007.366913]

Huber M F, Bailey T, Durrant-Whyte H and Hanebeck U D. 2008. On entropy approximation for Gaussian mixture random vectors//Proceedings of 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems. Seoul, Korea (South): IEEE: 181-188 [DOI: 10.1109/mfi.2008.4648062http://dx.doi.org/10.1109/mfi.2008.4648062]

Kopf J, Cohen M F, Lischinski D and Uyttendaele M. 2007. Joint bilateral upsampling//Proceedings of ACM SIGGRAPH 2007 Papers. San Diego, USA: ACM [DOI: 10.1145/1275808.1276497http://dx.doi.org/10.1145/1275808.1276497]

Kampa K, Hasanbelliu E and Principe J C. 2011. Closed-form Cauchy-Schwarz PDF divergence for mixture of Gaussians//Proceedings of 2011 International Joint Conference on Neural Networks. San Jose, USA: IEEE: 2578-2585. [DOI: 10.1109/IJCNN.2011.6033555http://dx.doi.org/10.1109/IJCNN.2011.6033555]

Li Y B, Feng J, Zhang H X and Li C Q. 2013. New algorithm of depth hole filling based on intensive bilateral filter. Industrial Control Computer, 26(11): 105-106, 109

李应彬, 冯杰, 张华熊, 李晨勤. 2013. 基于改进双边滤波的Kinect深度图像空洞修复算法研究. 工业控制计算机, 26(11): 105-106, 109 [DOI: 10.3969/j.issn.1001-182X.2013.11.046]

Lin L, Chen Y J and Guo T H. 2019. Kinect depth image restoration algorithm based on space-time domain data fusion. Science Technology and Engineering, 19(30): 215-220

林玲, 陈姚节, 郭同欢. 2019. 基于时空域数据融合的Kinect深度图像修复算法. 科学技术与工程, 19(30): 215-220 [DOI: 10.3969/j.issn.1671-1815.2019.30.032]

Liu J Y and Gong X J. 2013. Guided depth enhancement via anisotropic diffusion//Proceedings of the 14th Pacific-Rim Conference on Multimedia. Nanjing, China: Springer: 408-417 [DOI: 10.1007/978-3-319-03731-8_38http://dx.doi.org/10.1007/978-3-319-03731-8_38]

Matyunin S, Vatolin D, Berdnikov Y and Smirnov M. 2011. Temporal filtering for depth maps generated by Kinect depth camera//Proceedings of 2011 3DTV Conference: the True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON). Antalya, Turkey: IEEE: 1-4 [DOI: 10.1109/3dtv.2011.5877202http://dx.doi.org/10.1109/3dtv.2011.5877202]

Min D B, Lu J B and Do M N. 2012. Depth video enhancement based on weighted mode filtering. IEEE Transactions on Image Processing, 21(3): 1176-1190 [DOI: 10.1109/tip.2011.2163164]

Pertuz S and Kamarainen J. 2017. Region-based depth recovery for highly sparse depth maps//Proceedings of 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China: IEEE: 2074-2078 [DOI: 10.1109/icip.2017.8296647http://dx.doi.org/10.1109/icip.2017.8296647]

Scharstein D and Pal C. 2007. Learning conditional random fields for stereo//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, USA: IEEE: 1-8 [DOI: 10.1109/cvpr.2007.383191http://dx.doi.org/10.1109/cvpr.2007.383191]

Telea A. 2004. An image inpainting technique based on the fast marching method. Journal of Graphics Tools, 9(1): 23-34 [DOI: 10.1080/10867651.2004.10487596]

Lee K and Bresler Y. 2010. ADMiRA: atomic decomposition for minimum rank approximation. IEEE Transactions on Information Theory, 56(9): 4402-4416

Wang Z Y, Song X W, Wang S Z, Xiao J, Zhong R and Hu R M. 2016. Filling kinect depth holes via position-guided matrix completion. Neurocomputing, 215: 48-52 [DOI: 10.1016/j.neucom.2015.05.146]

Zeng X J and Lu C. 2013. The application of MRF based-on chaos-PSO optimization in depth information estimation. Journal of Huazhong University of Science and Technology (Natural Science Edition), 41(Z1): 223-225

曾祥进, 卢成. 2013. 混沌PSO优化的马尔可夫随机场的深度恢复. 华中科技大学学报(自然科学版), 41(Z1): 223-225 [DOI: 10.13245/j.hust.2013.s1.059]

Zhang Y D and Funkhouser T. 2018. Deep depth completion of a single RGB-D image//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 175-185 [DOI: 10.1109/cvpr.2018.00026http://dx.doi.org/10.1109/cvpr.2018.00026]

Zheng C Y, Li L F, Xiao Z S and Lu C. 2016. A depth image enhancement algorithm based on improved anisotropic diffusion. Computer Engineering and Science, 38(9): 1823-1829

郑传远, 李良福, 肖樟树, 陆铖. 2016. 一种改进的各向异性扩散深度图像增强算法. 计算机工程与科学, 38(9): 1823-1829 [DOI: 10.3969/j.issn.1007-130X.2016.09.013]

Alert me when the article has been cited

提交

An image segmentation model in combination with dissimilarity criterion and entropy rate super-pixel

Geometric active contour tracking based on locally model matching

Foreground discrimination in local model-matching tracking

Clothing retrieval combining hierarchical over-segmentation and cross-domain dictionary learning