浏览全部资源
扫码关注微信
纸质出版日期： 2021-04-16 ，
录用日期： 2020-10-07
移动端阅览
吴迪, 张旭东, 张骏, 范之国, 孙锐. 3D遮挡模型引导的光场图像深度获取[J]. 中国图象图形学报, 2021,26(4):924-938.
Di Wu, Xudong Zhang, Jun Zhang, Zhiguo Fan, Rui Sun. Light field depth estimation guided by 3D occlusion model[J]. Journal of Image and Graphics, 2021,26(4):924-938.
目的
2
光场相机可以通过单次曝光同时从多个视角采样单个场景，在深度估计领域具有独特优势。消除遮挡的影响是光场深度估计的难点之一。现有方法基于2D场景模型检测各视角遮挡状态，但是遮挡取决于所采样场景的3D立体模型，仅利用2D模型无法精确检测，不精确的遮挡检测结果将降低后续深度估计精度。针对这一问题，提出了3D遮挡模型引导的光场图像深度获取方法。
方法
2
向2D模型中的不同物体之间添加前后景关系和深度差信息，得到场景的立体模型，之后在立体模型中根据光线的传输路径推断所有视角的遮挡情况并记录在遮挡图（occlusion map）中。在遮挡图引导下，在遮挡和非遮挡区域分别使用不同成本量进行深度估计。在遮挡区域，通过遮挡图屏蔽被遮挡视角，基于剩余视角的成像一致性计算深度；在非遮挡区域，根据该区域深度连续特性设计了新型离焦网格匹配成本量，相比传统成本量，该成本量能够感知更广范围的色彩纹理，以此估计更平滑的深度图。为了进一步提升深度估计的精度，根据遮挡检测和深度估计的依赖关系设计了基于最大期望（exception maximization，EM）算法的联合优化框架，在该框架下，遮挡图和深度图通过互相引导的方式相继提升彼此精度。
结果
2
实验结果表明，本文方法在大部分实验场景中，对于单遮挡、多遮挡和低对比度遮挡在遮挡检测和深度估计方面均能达到最优结果。均方误差（mean square error，MSE）对比次优结果平均降低约19.75%。
结论
2
针对遮挡场景的深度估计，通过理论分析和实验验证，表明3D遮挡模型相比传统2D遮挡模型在遮挡检测方面具有一定优越性，本文方法更适用于复杂遮挡场景的深度估计。
Objective
2
Depth estimation from multiple images is a central task in computer vision. Reliable depth information provides an effective source for visual tasks
such as target detection
image segmentation
and special effects for movies. As one of the new multi-view image acquisition devices
the light field camera makes it more convenient to acquire multiple image data. A light field camera can simultaneously sample a scene from multiple viewpoints with a single exposure
which has unique advantages in portability and depth accuracy over other depth sensors. Occlusion is a challenging issue for light field depth estimation. For a non-occluded pixel on Lambertian surfaces
the angular patch corresponding to this pixel exhibits photo-consistency when refocused to its correct depth. However
the occluder will prevent viewpoints from sampling the same point. Thus
the photo-consistency fails to hold at occluded pixels. If the occluded viewpoints are accurately excluded
the photo-consistency of the remaining viewpoints can still be guaranteed. Therefore
how to identify the occluded viewpoints in the angular patch is crucial for accurate depth estimation. Previous works detected occlusion on the basis of the 2D model (RGB image) of the scene. However
occlusion is determined by the scene's 3D model
and it cannot be accurately detected using only the 2D model. Inaccurate occlusion detection will lead to low quality of depth estimation. In this study
we present a light field depth estimation algorithm that is robust to occlusion.
Method
2
First
we reconstruct the 3D scene model by adding the foreground-background relation and depth difference between different objects in the 2D model. On the basis of the 3D model
we directly calculate the occlusion state of each view and record it in the occlusion map. Further analysis demonstrates that the generated occlusion map can exclude all occluded viewpoints. Thanks to the occlusion map
the scene is able to be divided into occluded and non-occluded regions
so that more appropriate cost function can be adopted in different regions. In this study
if a spatial point is visible in a subset of viewpoints
this spatial point will be included in the occluded region. The remaining spatial points will be included in the non-occluded regions. In the occluded regions
we exclude the occluded viewpoints by the occlusion map and build the cost volume on the basis of the photo-consistency of the remaining viewpoints. In the non-occluded regions
on the basis of the depth continuity of these regions
we design a defocus grid matching cost function that captures textures over a wider area than traditional methods. A wider capture range means that our cost function is capable of collecting more information to increase its robustness. To propagate the effective information of higher confidence points to low confidence points
every slice in the final data cost volume is filtered using the edge-preserving filter. Compared with graph-based optimization
the filter-based method is more efficient and easy to parallelize. Moreover
because our occlusion map has excluded the possible occlusions
the filter-based method is enough for most examples. The initial disparity label is generated from the filtered cost volume using the winner-takes-all method. Finally
we exploit the dependence between the occlusion map and the depth map to further improve the accuracy of depth estimation. That is
the depth map can help the reconstruction of the 3D model required for occlusion detection
and the occlusion map can help the cost function exclude the occluded viewpoints. On the basis of this dependence
we integrate occlusion detection and depth estimation into an expectation-maximization-based optimization framework to alternatively improve the accuracy of the occlusion map and the depth map.
Result
2
Experiments are conducted on the HCI (Heidelberg Collaboratory for Image Processing) synthetic dataset and Stanford Lytro Illum dataset for real scenes. To ensure fairness
the number of depth labels of all cost-volume-based algorithms is uniformly set to 75. For quantitative evaluation
we use the percentage of bad pixels and the mean square error to measure the pros and cons of every algorithm. We also compare our occlusion detection method with state-of-the-art methods. Instead of evaluating the occlusion map of a single angular patch
we evaluate the occlusion map of all angular patches around the occlusion boundary. This evaluation method requires the algorithm to respond correctly to all degrees of occlusion. The experimental results show that the proposed method achieves better performance than other state-of-the-art methods in terms of both occlusion detection and depth estimation for single occlusion
multi-occlusion
and low-contrast occlusion. Compared with the suboptimal method
our mean square error is reduced by about 19.75% on average.
Conclusion
2
For the depth estimation of scenes with occlusion
the superiority of the proposed 3D occlusion model is demonstrated through theoretical analysis and experimental verification. The proposed depth estimation algorithm is more suitable for scenes with complex occlusion.
光场深度估计3D遮挡模型抗遮挡最大期望(EM)
light fielddepth estimation3D occlusion modelanti-occlusionexception maximization(EM)
Bolles R C, Baker H H, Marimont D H. 1987. Epipolar-plane image analysis: an approach to determining structure from motion. International Journal of Computer Vision, 1(1): 7-55[DOI:10.1007/bf00128525]
Boykov Y, Veksler O, Zabih R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11): 1222-1239[DOI:10.1109/34.969114]
Chen C, Lin H T, Yu Z, Kang S B, Yu J Y. 2014. Light field stereo matching using bilateral statistics of surface cameras//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 1518-1525[DOI: 10.1109/CVPR.2014.197http://dx.doi.org/10.1109/CVPR.2014.197]
Chen J, Hou J H, Ni Y, Chau L P. 2018. Accurate light field depth estimation with superpixel regularization over partially occluded regions. IEEE Transactions on Image Processing, 27(10): 4889-4900[DOI:10.1109/TIP.2018.2839524]
Guo Z H, Wu J L, Chen X F, Ma S, Zhu L C, Yang P, Xu B. 2019. Accurate light field depth estimation using multi-orientation partial angular coherence. IEEE Access, 7: 169123-169132[DOI:10.1109/ACCESS.2019.2954892]
Honauer K, Johannsen O, Kondermann D, Goldluecke B.2016. A dataset and evaluation methodology for depth estimation on 4D light fields//Proceedings of the 13th Asian Conference on Computer Vision. Taipei, China: Springer: 19-34[DOI: 10.1007/978-3-319-54187-7_2http://dx.doi.org/10.1007/978-3-319-54187-7_2]
Jeon H G, Park J, Choe G, Park J, Bok Y, Tai Y W, Kweon I S. 2015. Accurate depth map estimation from a lenslet light field camera//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 1547-1555[10.1109/CVPR.2015.7298762]
Johannsen O, Honauer K, Goldluecke B, Alperovich A, Battisti F, Bok Y, Brizzi M, Carli M, Choe G, Diebold M, Gutsche M, Jeon H G, Kweon I S, Park J, Park J, Schilling H, Sheng H, Si L P, Strecke M, Sulc A, Tai Y W, Wang Q, Wang T C, Wanner S, Xiong Z, Yu J Y, Zhang S, Zhu H. 2017. A taxonomy and evaluation of dense light field depth estimation algorithms//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, USA: IEEE: 1795-1812[DOI: 10.1109/CVPRW.2017.226http://dx.doi.org/10.1109/CVPRW.2017.226]
Lin H T, Chen C, Kang S B, Yu J Y. 2015. Depth recovery from light field using focal stack symmetry//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 3451-3459[DOI: 10.1109/ICCV.2015.394http://dx.doi.org/10.1109/ICCV.2015.394]
Ng R, Levoy M, Brédif M, Duval G, Horowitz M, Hanrahan P. 2005. Light field photography with a hand-held plenoptic camera. CSTR 2005-02. Stanford University Computer Science: 1-11
Rhemann C, Hosni A, Bleyer M, Rother C, Gelautz M. 2011. Fast cost-volume filtering for visual correspondence and beyond//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition 2011. Providence, USA: IEEE: 3017-3024[DOI: 10.1109/CVPR.2011.5995372http://dx.doi.org/10.1109/CVPR.2011.5995372]
Sheng H, Zhang S, Cao X C, Fang Y J, Xiong Z. 2017. Geometric occlusion analysis in depth estimation using integral guided filter for light-field image. IEEE Transactions on Image Processing, 26(12): 5758-5771[DOI:10.1109/TIP.2017.2745100]
Strecke M, Alperovich A, Goldluecke B. 2017. Accurate depth and normal maps from occlusion-aware focal stack symmetry//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2529-2537[DOI: 10.1109/CVPR.2017.271http://dx.doi.org/10.1109/CVPR.2017.271]
Tao M W, Hadap S, Malik J, Ramamoorthi R. 2013. Depth from combining defocus and correspondence using light-field cameras//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 673-680[DOI: 10.1109/ICCV.2013.89http://dx.doi.org/10.1109/ICCV.2013.89]
Wang T C, Efros A A, Ramamoorthi R. 2015. Occlusion-aware depth estimation using light-fieldcameras//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 3487-3495[DOI: 10.1109/ICCV.2015.398http://dx.doi.org/10.1109/ICCV.2015.398]
Wanner S, Goldluecke B. 2012. Globally consistent depth labeling of 4D light fields//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA: IEEE: 41-48[DOI: 10.1109/CVPR.2012.6247656http://dx.doi.org/10.1109/CVPR.2012.6247656]
Wanner S, Meister S, Goldluecke B. 2013. Datasets and benchmarks for densely sampled 4D light fields//Proceedings of the 18th International Workshop on Vision, Modeling and Visualization. Lugano, Switzerland: The Eurographics Association: 225-226[DOI: 10.2312/PE.VMV.VMV13.225-226http://dx.doi.org/10.2312/PE.VMV.VMV13.225-226]
Williem W, Park I K. 2016. Robust light field depth estimation for noisy scene with occlusion//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 4396-4404[DOI: 10.1109/CVPR.2016.476http://dx.doi.org/10.1109/CVPR.2016.476]
Williem W, Park I K, Lee K M. 2018. Robust light field depth estimation using occlusion-noise aware data costs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10): 2484-2497[DOI:10.1109/TPAMI.2017.2746858]
Zhang S, Sheng H, Li C, Zhang J, Xiong Z. 2016. Robust depth estimation for light field via spinning parallelogram operator. Computer Vision and Image Understanding, 145: 148-159[DOI:10.1016/j.cviu.2015.12.007]
Zhang S, Sheng H, Yang D, Zhang J, Xiong Z. 2018. Micro-lens-based matching for scene recovery in lenslet cameras. IEEE Transactions on Image Processing, 27(3): 1060-1075[DOI:10.1109/TIP.2017.2763823]
相关作者
相关机构
京公网安备11010802024621