浏览全部资源
扫码关注微信
Received：13 January 2020，
Revised：2020-3-22，
Accepted：29 March 2020，
Published：16 December 2020
移动端阅览
显著性检测一直是计算机视觉领域的关键问题，在视觉跟踪、图像压缩和目标识别等方面有着非常重要的应用。基于传统RGB图像和RGB-D（RGB depth）图像的显著性检测易受复杂背景、光照、遮挡等因素影响，在复杂场景的检测精度较低，鲁棒的显著性检测仍存在很大挑战。随着光场成像技术的发展，人们开始从新的途径解决显著性检测问题。光场数据记录着空间光线位置信息和方向信息，隐含场景的几何结构，能为显著性检测提供可靠的背景、深度等先验信息。因此，利用光场数据进行显著性检测得到了广泛关注，成为研究热点。尽管基于光场数据的显著性检测算法陆续出现，但是缺少对该问题的深刻理解以及研究进展的全面综述。本文系统地综述了基于光场数据的显著性检测研究现状，并进行深入探讨和展望。对光场理论以及用于光场显著性检测的公共数据集进行介绍；系统地介绍了光场显著性检测领域的算法模型和最新进展，从人工设计光场特征、稀疏编码特征和深度学习特征等方面进行全面阐述及分析；通过4个公共光场显著性数据集上的实验数据对不同方法的优缺点进行比较和分析，并结合实际应用指出当前研究的局限性与发展趋势。
Saliency detection is an important task in the computer vision community
especially in visual tracking
image compression
and object recognition tasks. However
the extant saliency detection methods based on RGB or RGB depth (RGB-D) often suffer from problems related to complex backgrounds
illumination
occlusion
and other factors
thereby leading to their inferior detection performance. In this case
a solution for improving the robustness of saliency detection results is warranted. In recent years
commercial and industrial light field cameras based on micro-lens arrays inserted between the main lens and the photosensor have introduced a new method for solving the saliency detection problem. The light field not only records spatial information but also the directions of all incoming light rays. The spatial and angular information inherent in a light field implicitly contains the geometry and reflection characteristics of the observed scene
which can provide reliable prior information for saliency detection
such as background clues and depth information. For example
the digital refocus technique can divide the light field into focal slices that focus at different depths. The background clues can be obtained from the focused areas. The light field contains effective saliency object occlusion information. Depth information also be obtained from the light field in various ways. Therefore
using light fields offers many advantages in dealing with problems related to saliency detection. Although saliency detection based on light fields has received much attention in recent years
a deep understanding of this method is yet to be achieved. In this paper
we review the research progress on light field saliency detection to build a foundation for future studies on this topic. First
we briefly discuss light field imaging theory
light field cameras
and the existing light field datasets used for saliency detection and then point out the differences among various datasets. Second
we systematically review the extant algorithms and the latest progress in light filed saliency detection from the aspects of hand-crafted features
sparse coding
and deep learning. Saliency detection algorithms based on light field hand-crafted features are generally based on the idea of contrast. These algorithms detect salient regions by calculating the feature difference between each pixel and super pixel as well as between other pixels and other super pixels. Saliency detection based on sparse coding and deep learning follow the same idea of feature learning
that is
they use image feature coding or the outstanding feature representation abilities of convolution network to determine the salient regions. By analyzing the experimental data on four publicly available light field saliency detection datasets
we summarize the advantages and disadvantages of the existing light field saliency detection methods
summarize the recent progress in light-field-based saliency detection and point out the limitations in this field. Only a few light field datasets are presently available for saliency detection
and these datasets are all generated by light field cameras based on micro-lens array
which has a narrow baseline. Therefore
the effective utilization of various information present in a light field remains a challenge. While saliency detection algorithms based on light fields have been proposed in previous studies
saliency detection based on light fields warrant further study due to the complexity of real scenes.
Achanta R, Hemami S, Estrada F and Süsstrunk S. 2009. Frequency-tuned salient region detection//Proceedings of 2009 IEEE International Conference on Computer Vision and Pattern Recognition. Miami, USA: IEEE: 1597-1604[ DOI: 10.1109/CVPR.2009.5206596 http://dx.doi.org/10.1109/CVPR.2009.5206596 ]
Adelson E H and Wang J Y A. 1992. Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):99-106[DOI:10.1109/34.121783]
Alexe B, Deselaers T and Ferrari V. 2012. Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2189-2202[DOI:10.1109/TPAMI.2012.28]
Bolles R C, Baker H H and Marimont D H. 1987. Epipolar-plane image analysis:an approach to determining structure from motion. International Journal of Computer Vision, 1(1):7-55[DOI:10.1007/BF00128525]
Borji A, Cheng M M, Jiang H Z and Li J. 2015. Salient object detection:a benchmark. IEEE Transactions on Image Processing, 24(12):5706-5722[DOI:10.1109/TIP.2015.2487833]
Borji A and Tanner J. 2016. Reconciling saliency and object center-bias hypotheses in explaining free-viewing fixations. IEEE Transactions on Neural Networks and Learning Systems, 27(6):1214-1226[DOI:10.1109/TNNLS.2015.2480683]
Chen L C, Papandreou G, Kokkinos I, Murphy K and Yuille A L. 2018. DeepLab:semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-848[DOI:10.1109/TPAMI.2017.2699184]
Cheng M M, Mitra N J, Huang X L, Torr P H S and Hu S M. 2015. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):569-582[DOI:10.1109/TPAMI.2014.2345401]
Georgiev T and Lumsdaine A. 2009. Resolution in plenoptic cameras//Proceedings of Computational Optical Sensing and Imaging. San Jose, California United States: Optical Society of America: CTuB3[ DOI: 10.1364/COSI.2009.CTuB3 http://dx.doi.org/10.1364/COSI.2009.CTuB3 ]
Georgiev T, Lumsdaine A and Goma S. 2009. High dynamic range image capture with plenoptic 2.0 camera//Proceedings of Signal Recovery and Synthesis 2009. San Jose, USA: Optical Society of America: SWA7P[ DOI: 10.1364/SRS.2009.SWA7P http://dx.doi.org/10.1364/SRS.2009.SWA7P ]
Georgiev T, Yu Z, Lumsdaine A and Goma S. 2013. Lytro camera technology: theory, algorithms, performance analysis//Proceedings of Multimedia Content and Mobile Devices. Burlingame, USA: SPIE: 86671J[ DOI: 10.1117/12.2013581 http://dx.doi.org/10.1117/12.2013581 ]
Gershun A. 1939. The light field. Journal of Mathematics and Physics, 18(1/4):51-151[DOI:10.1002/sapm193918151]
Girshick R, Donahue J, Darrell T and Malik J. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 580-587[ DOI: 10.1109/CVPR.2014.81 http://dx.doi.org/10.1109/CVPR.2014.81 ]
Han S and Vasconcelos N. 2014. Object recognition with hierarchical discriminant saliency networks. Frontiers in Computational Neuroscience, 8:#109[DOI:10.3389/fncom.2014.00109]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Itti L. 2004. Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Transactions on Image Processing, 13(10):1304-1318[DOI:10.1109/TIP.2004.834657]
Itti L, Koch C and Niebur E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11):1254-1259[DOI:10.1109/34.730558]
Jeon H G, Park J, Choe G, Park J, Bok Y, Tai Y W and Kweon I S. 2019. Depth from a light field image with learning-based matching costs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):297-310[DOI:10.1109/TPAMI.2018.2794979]
Jiang H Z, Wang J D, Yuan Z J, Wu Y, Zheng N N and Li S P. 2013. Salient object detection: a discriminative regional feature integration approach//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE: 2083-2090[ DOI: 10.1109/CVPR.2013.271 http://dx.doi.org/10.1109/CVPR.2013.271 ]
Ju R, Ge L, Geng W J, Ren T W and Wu G S. 2014. Depth saliency based on anisotropic center-surround difference//Proceedings of 2014 IEEE International Conference on Image Processing. Paris, France: IEEE: 1115-1119[ DOI: 10.1109/ICIP.2014.7025222 http://dx.doi.org/10.1109/ICIP.2014.7025222 ]
Krähenbühl P and Koltun V. 2012. Efficientinference in fully connected CRFs with Gaussian edge potentials[EB/OL].[2020-02-13] . https://arxiv.org/pdf/1210.5644.pdf https://arxiv.org/pdf/1210.5644.pdf
Lang C Y, Nguyen T V, Katti H, Yadati K, Kankanhalli M and Yan S C. 2012. Depth matters: influence of depth cues on visual saliency//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy: Springer: 101-115[ DOI: 10.1007/978-3-642-33709-3_8 http://dx.doi.org/10.1007/978-3-642-33709-3_8 ]
Levoy M and Hanrahan P. 1996. Light field rendering//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. New Orleans, USA: ACM: 31-42[ DOI: 10.1145/237170.237199 http://dx.doi.org/10.1145/237170.237199 ]
Li G B and Yu Y Z. 2015a. Visual saliency based on multiscale deep features//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 5455-5463[ DOI: 10.1109/CVPR.2015.7299184 http://dx.doi.org/10.1109/CVPR.2015.7299184 ]
Li N Y, Sun B L and Yu J Y. 2015b. A weighted sparse coding framework for saliency detection//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 5216-5223[ DOI: 10.1109/CVPR.2015.7299158 http://dx.doi.org/10.1109/CVPR.2015.7299158 ]
Li N Y, Ye J W, Ji Y, Ling H B and Yu J Y. 2014. Saliency detection on light field//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 2806-2813[ DOI: 10.1109/CVPR.2014.359 http://dx.doi.org/10.1109/CVPR.2014.359 ]
Li X, Li Y, Shen C H, Dick A and Van Den Hengel A. 2013a. Contextual hypergraph modeling for salient object detection//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 3328-3335[ DOI: 10.1109/ICCV.2013.413 http://dx.doi.org/10.1109/ICCV.2013.413 ]
Li X H, Lu H C, Zhang L H, Ruan X and Yang M H. 2013b. Saliency detection via dense and sparse reconstruction//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 2976-2983[ DOI: 10.1109/ICCV.2013.370 http://dx.doi.org/10.1109/ICCV.2013.370 ]
Long J, Shelhamer E and Darrell T. 2015. Fully convolutional networks for semantic segmentation//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 3431-3440[ DOI: 10.1109/CVPR.2015.7298965 http://dx.doi.org/10.1109/CVPR.2015.7298965 ]
Mahadevan V and Vasconcelos N. 2012. On the connections between saliency and tracking//Proceedings of the 26th Annual Conference on Neural Information Processing Systems. Lake Tahoe, USA: Curran Associates: 1673-1681
Margolin R, Zelnik-Manor L and Tal A. 2014. How to evaluate foreground maps//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 248-255[ DOI: 10.1109/CVPR.2014.39 http://dx.doi.org/10.1109/CVPR.2014.39 ]
Marwah K, Wetzstein G, Bando Y and Raskar R. 2013. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, 32(4):#46[DOI:10.1145/2461912.2461914]
Moore C M, Elsinger C L and Lleras A. 2001. Visual attention and the apprehension of spatial relations:the case ofdepth. Perception and Psychophysics, 63(4):595-606[DOI:10.3758/BF03194424]
Ng R. 2006. Digital Light Field Photography. Stanford: Stanford University: 1-203
Ng R, Levoy M, Brédif M, Duval G, Horowitz M and Hanrahan P. 2005. Light field photography with a hand-held plenoptic camera. Stanford University and Duval Design: 1-11
Peng H W, Li B, Xiong W H, Hu W M and Ji R R. 2014. RGBD salient object detection: a benchmark and algorithms//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 92-109[ DOI: 10.1007/978-3-319-10578-9_7 http://dx.doi.org/10.1007/978-3-319-10578-9_7 ]
Peng J Y, Xiong Z W, Liu D and Chen X J. 2018. Unsupervised depth estimation from light field using a convolutional neural network//Proceedings of 2018 International Conference on 3D Vision. Verona, Italy: IEEE: 295-303[ DOI: 10.1109/3DV.2018.00042 http://dx.doi.org/10.1109/3DV.2018.00042 ]
Piao Y R, Li X and Zhang M. 2018. Depth-induced cellular automata for light field saliency//Proceedings of Frontiers in Optics. Washington DC, USA: Optical Society of America: FTh3E-3[ DOI: 10.1364/FIO.2018.FTh3E.3 http://dx.doi.org/10.1364/FIO.2018.FTh3E.3 ]
Piao Y R, Li X, Zhang M, Yu J Y and Lu H C. 2019a. Saliency detection via depth-induced cellular automata on light field. IEEE Transactions on Image Processing, 29:1879-1889[DOI:10.1109/TIP.2019.2942434]
Piao Y R, Rong Z K, Zhang M, Li X and Lu H C. 2019b. Deep light-field-driven saliency detection from a single view//Proceedings of the 28th International Joint Conference on Artificial Intelligence. Macao, China: IJCAI: 904-911[ DOI: 10.24963/ijcai.2019/127 http://dx.doi.org/10.24963/ijcai.2019/127 ]
Poggio G F and Poggio T. 1984. The analysis of stereopsis. Annual Review of Neuroscience, 7:379-412[DOI:10.1146/annurev.ne.07.030184.002115]
Ren J Q, Gong X J, Yu L, Zhou W H and Ying M Y. 2015. Exploiting global priors for RGB-D saliency detection//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston, USA: IEEE: 25-32[ DOI: 10.1109/CVPRW.2015.7301391 http://dx.doi.org/10.1109/CVPRW.2015.7301391 ]
Sheng H, Zhang S, Liu X Y and Xiong Z. 2016. Relative location for light field saliency detection//Proceedings of 2016 IEEE International Conference on Acoustics, Speech, and Signal Processing. Shanghai, China: IEEE: 1631-1635[ DOI: 10.1109/ICASSP.2016.7471953 http://dx.doi.org/10.1109/ICASSP.2016.7471953 ]
Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition[EB/OL].[2020-02-13] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf
Srinivasan P P, Wang T Z, Sreelal A, Ramamoorthi R and Ng R. 2017. Learning to synthesize a 4D RGBD light field from a single image//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 2262-2270[ DOI: 10.1109/ICCV.2017.246 http://dx.doi.org/10.1109/ICCV.2017.246 ]
Takahashi K, Kubota A and Naemura T. 2003. All in-focus view synthesis from under-sampled light fields//Proceeding of the 13th International Conference on Artificial Reality and Telexistence. Tokyo, Japan: ICAT: 249-256
Tang Y B, Wu X Q and Bu W. 2016. Deeply-supervised recurrent convolutional neural network for saliency detection//Proceedings of the 24th ACM International Conference on Multimedia. Amsterdam, The Netherlands: ACM: 397-401[ DOI: 10.1145/2964284.2967250 http://dx.doi.org/10.1145/2964284.2967250 ]
Tao M W, Srinivasan P P, Hadap S, Rusinkiewicz S, Malik J and Ramamoorthi R. 2017. Shape estimation from shading, defocus, and correspondence using light-field angular coherence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(3):546-560[DOI:10.1109/TPAMI.2016.2554121]
Tong N, Lu H C, Zhang Y and Ruan X. 2015. Salient object detection via global and local cues. Pattern Recognition, 48(10):3258-3267[DOI:10.1016/j.patcog.2014.12.005]
Vaish V, Wilburn B, Joshi N and Levoy M. 2004. Using plane + parallax for calibrating dense camera arrays//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC, USA: IEEE: #1315006[ DOI: 10.1109/CVPR.2004.1315006 http://dx.doi.org/10.1109/CVPR.2004.1315006 ]
Veeraraghavan A, Raskar R, Agrawal A, Mohan A and Tumblin J. 2007. Dappled photography:mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Transactions on Graphics, 26(3):#69[DOI:10.1145/1276377.1276463]
Wang A Z, Wang M H, Li X Y, Mi Z T and Zhou H. 2017. A two-stage Bayesian integration framework for salient object detection on light field. Neural Processing Letters, 46(3):1083-1094[DOI:10.1007/s11063-017-9610-x]
Wang H Q, Yan B, Wang X Z, Zhang Y B and Yang Y. 2018a. Accurate saliency detection based on depth feature of 3D images. Multimedia Tools and Applications, 77(12):14655-14672[DOI:10.1007/s11042-017-5052-8]
Wang S Z, Liao W J, Surman P, Tu Z G, Zheng Y J and Yuan J S. 2018b. Salience guided depth calibration for perceptually optimized compressive light field 3d display//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2031-2040[ DOI: 10.1109/CVPR.2018.00217 http://dx.doi.org/10.1109/CVPR.2018.00217 ]
Wang T C, Zhu J Y, Hiroaki E, Chandraker M, Efros A A and Ramamoorthi R. 2016. A 4D light-field dataset and CNN architectures for material recognition//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 121-138[ DOI: 10.1007/978-3-319-46487-9_8 http://dx.doi.org/10.1007/978-3-319-46487-9_8 ]
Wang T T, Piao Y R, Li H C, Li X and Zhang L H. 2019. Deep learning for light field saliency detection//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, South Korea: IEEE: 8837-8847[ DOI: 10.1109/ICCV.2019.00893 http://dx.doi.org/10.1109/ICCV.2019.00893 ]
Wanner S and Goldluecke B. 2012. Globally consistent depth labeling of 4D light fields//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA: IEEE: 41-48[ DOI: 10.1109/CVPR.2012.6247656 http://dx.doi.org/10.1109/CVPR.2012.6247656 ]
Wei Y C, Wen F, Zhu W J and Sun J. 2012. Geodesic saliency using background priors//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy: Springer: 29-42[ DOI: 10.1007/978-3-642-33712-3_3 http://dx.doi.org/10.1007/978-3-642-33712-3_3 ]
Wilburn B, Joshi N, Vaish V, Talvala E V, Antunez E, Barth A, Adams A, Horowitz M and Levoy M. 2005. High performance imaging using large camera arrays//Proceedings of ACM SIGGRAPH 2005. Los Angeles, California, USA: ACM: 765-776[ DOI: 10.1145/1186822.1073259 http://dx.doi.org/10.1145/1186822.1073259 ]
Williem, Park I K and Lee K M. 2018. Robust light field depth estimation using occlusion-noise aware data costs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2484-2497[DOI:10.1109/TPAMI.2017.2746858]
Wolfe J M and Horowitz T S. 2004. What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience, 5(6): 495-501[ DOI: 10.1038/nrn1411 http://dx.doi.org/10.1038/nrn1411 ]
Xu L F, Li H L, Zeng L Y and Ngan K N. 2013. Saliency detection using joint spatial-color constraint and multi-scale segmentation. Journal of Visual Communication and Image Representation, 24(4):465-476[DOI:10.1016/j.jvcir.2013.02.007]
Yang J C, Everett M, Buehler C and McMillan L. 2002. A real-time distributed light field camera//Debevec P and Gibson S, eds. Eurographics Workshop on Rendering.[s.l.]: The Eurographics Association: 77-86[ DOI: 10.2312/EGWR/EGWR02/077-086 http://dx.doi.org/10.2312/EGWR/EGWR02/077-086 ]
Yu Z, Guo X Q, Lin H B, Lumsdaine A and Yu J Y. 2013. Line assisted light field triangulation and stereo matching//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 2792-2799[ DOI: 10.1109/ICCV.2013.347 http://dx.doi.org/10.1109/ICCV.2013.347 ]
Zhang C and Chen T. 2004. A self-reconfigurable camera array//Proceedings of ACM SIGGRAPH 2004.[s.l.]: ACM: 243-254[ DOI: 10.1145/1186223.1186412 http://dx.doi.org/10.1145/1186223.1186412 ]
Zhang J, Liu Y M, Zhang S P, Poppe R and Wang M. 2020. Light field saliency detection with deep convolutional networks. IEEE Transactions on Image Processing, 29:4421-4434[DOI:10.1109/TIP.2020.2970529]
Zhang J, Wang M, Gao J, Wang Y, Zhang X D and Wu X D. 2015. Saliency detection with a deeper investigation of light field//Proceedings of the 24th International Joint Conference on Artificial Intelligence. Buenos Aires, Argentina: IJCAI: 2212-2218
Zhang J, Wang M, Lin L, Yang X, Gao J and Rui Y. 2017. Saliency detection on light field:a multi-cue approach. ACM Transactions on Multimedia Computing, Communications, and Applications, 13(3):#32[DOI:10.1145/3107956]
Zhang M, Li J J, Wei J, Piao Y R and Lu H C. 2019. Memory-oriented decoder for light field salient object detection//Proceedings of the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada: [s.n.]: 896-906
Zhang Z Y. 2012. Microsoft kinect sensor and its effect. IEEE Multimedia, 19(2):4-10[DOI:10.1109/MMUL.2012.24]
Zhao Q and Koch C. 2012. Learning visual saliency by combining feature maps in a nonlinear manner using AdaBoost. Journal of Vision, 12(6):22-22[DOI:10.1167/12.6.22]
相关文章
相关作者
相关机构
京公网安备11010802024621
