浏览全部资源
扫码关注微信
Published： 16 May 2023 ，
移动端阅览
蔡泽民， 廖小鑫， 赖剑煌， 陈军. 2023. 结构纹理感知下的鲁棒光流估计及人脸活检应用. 中国图象图形学报， 28(05):1445-1461
Cai Zemin， Liao Xiaoxin， Lai Jianhuang， Chen Jun. 2023. Robust optical flow estimation method based on structure-texture aware retinex model and its application on face anti-spoofing. Journal of Image and Graphics， 28(05):1445-1461
目的
2
光流估计是计算机视觉研究中的一个重要方向，尽管光流估计方法不断改进，但光照变化条件下光流计算精度的提高仍然是一个尚待解决的挑战。人脸反欺诈方法对于确保人脸识别系统的安全性十分重要，光照鲁棒的脸部运动光流特征能为人脸活体检测提供有关运动和结构的可靠信息。为了获得对含光照变化视频中物体运动的理解能力并应用于人脸活体检测，提高系统性能，提出了一种基于结构纹理感知视网膜模型的鲁棒光流估计方法。
方法
2
基于Retinex理论，通过结构纹理感知方式将图像中的反射分量与光照分量充分解耦。由于反射分量具有丰富的纹理信息且光照分量中包含部分有用的结构信息，因此对所提取的光照分量进行滤波操作后再与反射分量一起融合到光流模型中，有效提高了光流估计的鲁棒性。为使模型所获光流具有更好的边缘保持性，采用光滑—稀疏正则化约束方式进行最小化求解。本文给出了求解优化问题的数值方法。
结果
2
采用MPI Sintel数据集图像序列，与PWC-Net、DCFlow+KF和FDFlowNet（fast deep flownet）等主流算法进行对比实验，本文方法在Clean和Final数据集中均得到最低的平均终点误差（end-point error，EPE），分别为2.473和4.807，在3个公开数据集上进行的评测进一步验证了本文方法的鲁棒性。最后，将所提取的脸部运动光流特征在人脸反欺诈数据集上进行了活体检测对比实验，对比实验结果验证了提出的光流估计算法更具鲁棒性，改善了人脸活体检测的效果。
结论
2
提出的光流计算模型，在不同光照变化条件下具有良好的鲁棒性，更适合于人脸活体检测应用。本项目代码链接为
https：//github.com/Xiaoxin-Liao/STARFlow
https://github.com/Xiaoxin-Liao/STARFlow
。
Objective
2
Optical flow estimation is essential for computer vision and image processing， which is focused on pixel-wise motions between consecutive images. It is beneficial for multiple research domains like target tracking， crowd flow segmentation， and human behavior analysis. In addition， optical flow can be as an effective tool for video-based motion extraction. Hence， it was investigated to find out the face spoofing-relevant motion clues. It is still challenged for accurate optical flow estimation under complex illumination circumstances although optical flow estimation has attracted wide attention in the field of computer vision. Most of dense optical flow estimation methods are based on variational iteration in terms of Horn and Schunck’s framework. The variational framework consists of data and regularization terms in common. Due to the uncontrolled illumination conditions are required to be clarified， the gradient constancy hypothesis can be adopted as the data term for variational model. The brightness constancy assumption can be optimized to a certain extent. However， neither brightness constancy assumption nor gradient constancy hypothesis effective in representing the complex illumination variations. As a result， the calculation of optical flow is mutual-benefited to illumination changes. Optical flow estimations are required to be regularized because of the inherent ill-posedness. The Tikhonov regularization is often based on the L2 norm， and it is minimized in terms of small-amplitude coefficients-distributed preservation， which can capture global patterns well in optical flow calculation. And， the minimization of the L1 norm is focused on more zero- or small-amplitude coefficients， and less large-amplitude ones. For discrete signals， L1 norm can give out better results than L2. L1 norm-based optical flow computation can be more precisely for mathematical modeling. To deal with regularization problems， conventional L1 and L2 norm is still related to current variational optical flow models. Robust penalty function-relevant L1 or L2 norm regularization can be used to generate smooth flow field and preserve motion inconsistency. However， it could lose the fine-scale motion structures and produce excessive segmentation artifacts. The image pre-processing methods can also be carried out in estimation of optical flow while illumination changes cannot be ignored between two consecutive frames. Optical flow calculation can provide motion features in facial biometric systems as well， which is concerned for such domains like surveillance， access control and forensic investigations. However， one of the challenges of facial biometric systems is the high possibility of the systems being deceived or spoofed by non-real faces. Recent face anti-spoofing technique can be as an effective pathway for facial biometric systems.
Method
2
to strengthen video-based targets motion in relevance with illumination changes， our research is focused on structure-texture-perceptive retinex model and optical flow-robust estimation for human-facial anti-spoofing application. For retinex theory， to improve robustness of optical flow against non-uniform illumination， the components of illumination and reflectance in the image are separated by decoupling. Reflection and illumination components are separated from the image base on a stronger reflectivity constancy assumption. After that， the illumination component is filtered through a low pass filter and it is then integrated into the new optical flow model. Additionally， a smooth-sparse regularization constraint is adopted to preserve edges and improve the accuracy of optical flow estimation. Furthermore， the numerical implementation of the model is demonstrated.
Result
2
Comparative analysis is carried out with some state-of-the-art optical flow estimation methods， including the variational based methods and deep learning based approaches on 3 publicity datasets of Middleburry， MPI Sintel and KITTI 2015. The quantitative analysis is carried out in terms of average angular error （AAE）， average end-point error （EPE） and the Fl scores in comparison with other related optical flow computations. Moreover， to evaluate the robustness of optical flow with respect to variations of illumination using the datasets， we consider conducting simulations of illumination to the source images. To render synthetic images， illumination patterns of linear， sinusoidal， Gaussian and mixture of Gaussian are involved in. To simulate the regularity and variability of real-world illumination patterns， such parameters like additive factor， multiplicative factor and gamma correlation are used as well. The experimental results show that our model outperforms all other evaluated methods on the three public datasets and their synthetic versions with different illumination patterns. To verify the feasibility of illumination-invariant method， the calculation is applied to obtain human-facial optical flow motion features， and face liveness-detected experiments are conducted on the Institute of Automation， Chinese Academy of Sciences （CASIA） face anti-spoofing database. The proposed STARFlow method is compared to some popular anti-spoofing methods related to optical flow like weighted regularization transform （WRT） and ARFlow. The quantitative and comparative evaluation metrics is composed of the accuracy of face anti-spoofing classification and the half total error rate （HTER）. Similarly， to validate the illumination robustness of the proposed method under challenging illumination changes， four synthetic illumination patterns are also appended to the dataset. Experiments are carried out on the basis of the completed CASIA database and the four illumination environments.
Conclusion
2
In this study， a new variational optical flow estimation model is facilitated in terms of structure-texture aware retinex theory. Experimental results validate that the proposed model outperforms several state-of-the-art optical flow estimation approaches， including some variational based methods and deep learning based methods. Meanwhile， the proposed STARFlow method can achieve a potential illumination-invariance in terms of face anti-spoofing under different illumination changes circumstances. The source code of this project is available at：
https：//github.com/Xiaoxin-Liao/STARFlow
https://github.com/Xiaoxin-Liao/STARFlow
.
Retinex模型结构纹理感知光照变化L0范数正则化光流人脸反欺诈
Retinex modelstructure-texture awarenessvarying illuminationL0 gradient regularizationoptical flowface anti-spoofing
Bailer C， Taetz B and Stricker D. 2019. Flow fields： dense correspondence fields for highly accurate large displacement optical flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence， 41（8）： 1879-1892 ［DOI： 10.1109/TPAMI.2018.2859970http://dx.doi.org/10.1109/TPAMI.2018.2859970］
Baker S， Scharstein D， Lewis J P， Roth S， Black M J and Szeliski R. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision， 92（1）： 1-31 ［DOI： 10.1007/s11263-010-0390-2http://dx.doi.org/10.1007/s11263-010-0390-2］
Bao W B， Zhang X Y， Chen L and Gao Z Y. 2019. KalmanFlow 2.0： efficient video optical flow estimation via context-aware Kalman filtering. IEEE Transactions on Image Processing， 28（9）： 4233-4246 ［DOI： 10.1109/TIP.2019.2903656http://dx.doi.org/10.1109/TIP.2019.2903656］
Brox T， Bruhn A， Papenberg N and Weickert J. 2004. High accuracy optical flow estimation based on a theory for warping//Proceedings of the 8th European Conference on Computer Vision. Prague， Czech Republic： Springer： 25-36 ［DOI： 10.1007/978-3-540-24673-2_3http://dx.doi.org/10.1007/978-3-540-24673-2_3］
Bruhn A and Weickert J. 2005. Towards ultimate motion estimation： combining highest accuracy with real-time performance//Proceedings of the 10th International Conference on Computer Vision. Beijing， China： IEEE： 749-755 ［DOI： 10.1109/ICCV.2005.240http://dx.doi.org/10.1109/ICCV.2005.240］
Butler D J， Wulff J， Stanley G B and Black M J. 2012. A naturalistic open source movie for optical flow evaluation//Proceedings of the 12th European Conference on Computer Vision. Florence， Italy： Springer： 611-625 ［DOI： 10.1007/978-3-642-33783-3_44http://dx.doi.org/10.1007/978-3-642-33783-3_44］
Cai B L， Xu X M， Guo K L， Jia K， Hu B and Tao D C. 2017. A joint intrinsic-extrinsic prior model for retinex//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 4000-4009 ［DOI： 10.1109/ICCV.2017.431http://dx.doi.org/10.1109/ICCV.2017.431］
Chen J， Cai Z M， Lai J H and Xie X H. 2018. Fast optical flow estimation based on the split Bregman method. IEEE Transactions on Circuits and Systems for Video Technology， 28（3）： 664-678 ［DOI： 10.1109/TCSVT.2016.2615324http://dx.doi.org/10.1109/TCSVT.2016.2615324］
Chingovska I， Yang J， Lei Z， Yi D， Li S Z， Kahm O， Glaser C， Damer N， Kuijper A， Nouak A， Komulainen J， Pereira T， Gupta S， Khandelwal S， Bansal S， Rai A， Krishna T， Goyal D， Waris M A， Zhang H， Ahmad I， Kiranyaz S， Gabbouj M， Tronci R， Pili M， Sirena N， Roli F， Galbally J， Ficrrcz J， Pinto A， Pedrini H， Schwartz W S， Rocha A， Anjos A and Marcel S. 2013. The 2nd competition on counter measures to 2D face spoofing attacks//Proceedings of 2013 International Conference on Biometrics. Madrid， Spain： IEEE： 1-6 ［DOI： 10.1109/ICB.2013.6613026http://dx.doi.org/10.1109/ICB.2013.6613026］
Danon D， Averbuch-Elor H， Fried O and Cohen-Or D. 2019. Unsupervised natural image patch learning. Computational Visual Media， 5（3）： 229-237 ［DOI： 10.1007/s41095-019-0147-yhttp://dx.doi.org/10.1007/s41095-019-0147-y］
Drulea M and Nedevschi S. 2013. Motion estimation using the correlation transform. IEEE Transactions on Image Processing， 22（8）： 3260-3270 ［DOI： 10.1109/TIP.2013.2263149http://dx.doi.org/10.1109/TIP.2013.2263149］
Fang M， Xu J， Takauji H， Kaneko S I and Xu H H. 2013. Efficient computation of optical flow using complementary voting. Acta Automatica Sinica， 39（7）： 1080-1092
方明， 徐晶， 高氏秀则， 金子俊一， 徐洪华. 2013. 一种基于互补投票的高效光流计算方法. 自动化学报， 39（7）： 1080-1092 ［DOI： 10.3724/SP.J.1004.2013.01080http://dx.doi.org/10.3724/SP.J.1004.2013.01080］
Fathy M E， Tran Q H， Zia M Z， Vernaza P and Chandraker M. 2018. Hierarchical metric learning and matching for 2D and 3D geometric correspondences//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 832-850 ［DOI： 10.1007/978-3-030-01267-0_49http://dx.doi.org/10.1007/978-3-030-01267-0_49］
Fu J Y， Yu L， Yang W and Lu X. 2021. Event-based continuous optical flow estimation. Acta Automatica Sinica， 47： 1-12
付婧祎， 余磊， 杨文， 卢昕. 2021. 基于事件相机的连续光流估计. 自动化学报， 47： 1-12 ［DOI： 10.16383/j.aas.c210242http://dx.doi.org/10.16383/j.aas.c210242］
Geiger A， Lenz P， Stiller C and Urtasun R. 2013. Vision meets robotics： the KITTI dataset. The International Journal of Robotics Research， 32（11）： 1231-1237 ［DOI： 10.1177/0278364913491297http://dx.doi.org/10.1177/0278364913491297］
Horn B K and Schunck B G. 1981. Determining optical flow. Artificial Intelligence， 17（1-3）： 185-203 ［DOI： 10.1016/0004-3702（81）90024-2http://dx.doi.org/10.1016/0004-3702（81）90024-2］
Hui T W， Tang X O and Loy C C. 2018. LiteFlowNet： a lightweight convolutional neural network for optical flow estimation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 8981-8989 ［DOI： 10.1109/CVPR.2018.00936http://dx.doi.org/10.1109/CVPR.2018.00936］
Hur J and Roth S. 2021. Self-supervised multi-frame monocular scene flow//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 2683-2693 ［DOI： 10.1109/CVPR46437.2021.00271http://dx.doi.org/10.1109/CVPR46437.2021.00271］
Janai J， Güney F， Ranjan A， Black M and Geiger A. 2018. Unsupervised learning of multi-frame optical flow with occlusions//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 713-731 ［DOI： 10.1007/978-3-030-01270-0_42http://dx.doi.org/10.1007/978-3-030-01270-0_42］
Kong L T and Yang J. 2020. Fdflownet： fast optical flow estimation using a deep lightweight network//Proceedings of 2020 IEEE International Conference on Image Processing. Abu Dhabi， United Arab Emirates： IEEE： 1501-1505 ［DOI： 10.1109/ICIP40778.2020.9191101http://dx.doi.org/10.1109/ICIP40778.2020.9191101］
Kumar A， Tung F， Wong A and Clausi D A. 2013. A decoupled approach to illumination-robust optical flow estimation. IEEE Transactions on Image Processing， 22（10）： 4136-4147 ［DOI： 10.1109/TIP.2013.2270374http://dx.doi.org/10.1109/TIP.2013.2270374］
Land E H. 1977. The retinex theory of color vision. Scientific American， 237（6）： 108-128 ［DOI： 10.1038/scientificamerican1277-108http://dx.doi.org/10.1038/scientificamerican1277-108］
Liu L， Zhang J N， He R F， Liu Y， Wang Y B， Tai Y， Luo D H， Wang C J， Li J L and Huang F Y. 2020a. Learning by analogy： reliable supervision from transformations for unsupervised optical flow estimation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 6488-6497 ［DOI： 10.1109/cvpr42600.2020.00652http://dx.doi.org/10.1109/cvpr42600.2020.00652］
Liu WZ， Salzmann M and Fua P. 2020b. Estimating people flows to better count them in crowded scenes//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 723-740 ［DOI： 10.1007/978-3-030-58555-6_43http://dx.doi.org/10.1007/978-3-030-58555-6_43］
Luo C X， Yang Z H， Wang P， Wang Y， Xu W， Nevatia R and Yuille A. 2020. Every pixel counts++： joint learning of geometry and motion with 3D holistic understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence， 42（10）： 2624-2641 ［DOI： 10.1109/TPAMI.2019.2930258http://dx.doi.org/10.1109/TPAMI.2019.2930258］
Maurer D， Marniok N， Goldluecke B and Bruhn A. 2018. Structure-from-motion-aware PatchMatch for adaptive optical flow estimation//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 575-592 ［DOI： 10.1007/978-3-030-01237-3_35http://dx.doi.org/10.1007/978-3-030-01237-3_35］
Mei L， Lai J H， Xie X H， Zhu J Y and Chen J. 2020. Illumination-invariance optical flow estimation using weighted regularization transform. IEEE Transactions on Circuits and Systems for Video Technology， 30（2）： 495-508 ［DOI： 10.1109/TCSVT.2019.2890861http://dx.doi.org/10.1109/TCSVT.2019.2890861］
Mileva Y， Bruhn A and Weickert J. 2007. Illumination-robust variational optical flow with photometric invariants//Proceedings of the 29th Joint Pattern Recognition Symposium. Heidelberg， Germany： Springer： 152-162 ［DOI： 10.1007/978-3-540-74936-3_16http://dx.doi.org/10.1007/978-3-540-74936-3_16］
Molnár J， Chetverikov D and Fazekas S. 2010. Illumination-robust variational optical flow using cross-correlation. Computer Vision and Image Understanding， 114（10）： 1104-1114 ［DOI： 10.1016/j.cviu.2010.07.006http://dx.doi.org/10.1016/j.cviu.2010.07.006］
Schuster T， Wolf L and Gadot D. 2017. Optical flow requires multiple strategies （but only one network）//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 6921-6930 ［DOI： 10.1109/CVPR.2017.732http://dx.doi.org/10.1109/CVPR.2017.732］
Shao X Q， Yang Y and Liu Y L. 2021. Review of optical flow algorithms in fluid motion estimation. Journal of Image and Graphics， 26（2）： 355-367
邵绪强， 杨艳， 刘艺林. 2021. 流体运动估计光流算法研究综述. 中国图象图形学报， 26（2）： 355-367 ［DOI： 10.11834/jig.200050http://dx.doi.org/10.11834/jig.200050］
Solmaz B， Moore B E and Shah M. 2012. Identifying behaviors in crowd scenes using stability analysis for dynamical systems. IEEE Transactions on Pattern Analysis and Machine Intelligence， 34（10）： 2064-2070 ［DOI： 10.1109/TPAMI.2012.123http://dx.doi.org/10.1109/TPAMI.2012.123］
Sun D Q， Yang X D， Liu M Y and Kautz J. 2018. PWC-Net： CNNs for optical flow using pyramid， warping， and cost volume//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 8934-8943 ［DOI： 10.1109/CVPR.2018.00931http://dx.doi.org/10.1109/CVPR.2018.00931］
Tosi F， Aleotti F， Ramirez P Z， Poggi M， Salti S， Di Stefano L and Mattoccia S. 2020. Distilled semantics for comprehensive scene understanding from videos//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 4653-4664 ［DOI： 10.1109/CVPR42600.2020.00471http://dx.doi.org/10.1109/CVPR42600.2020.00471］
Wedel A， Pock T， Zach C， Bischof H and Cremers D. 2009. An improved algorithm for TV-L1 optical flow//Cremers D， Rosenhahn B， Yuille A L and Schmidt F R， eds. Statistical and Geometrical Approaches to Visual Motion Analysis. Berlin， Heidelberg， Germany： Springer： 23-45 ［DOI： 10.1007/978-3-642-03061-1_2http://dx.doi.org/10.1007/978-3-642-03061-1_2］
Wei L Y， Lefebvre S， Kwatra V and Turk G. 2009. State of the art in example-based texture synthesis//M and GreinerGPauly， eds. Eurographics 2009 — State of the Art Reports. ［s.l.］： The Eurographics Association： 93-117 ［DOI： 10.2312/egst.20091063http://dx.doi.org/10.2312/egst.20091063］
Wulff J， Sevilla-Lara L and Black M J. 2017. Optical flow in mostly rigid scenes//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 6911-6920 ［DOI： 10.1109/CVPR.2017.731http://dx.doi.org/10.1109/CVPR.2017.731］
Xu J， Hou Y K， Ren D W， Liu L， Zhu F， Yu M Y， Wang H Q and Shao L. 2020. STAR： a structure and texture aware retinex model. IEEE Transactions on Image Processing， 29： 5022-5037 ［DOI： 10.1109/TIP.2020.2974060http://dx.doi.org/10.1109/TIP.2020.2974060］
Xu J， Ranftl R and Koltun V. 2017. Accurate optical flow via direct cost volume processing//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 5807-5815 ［DOI： 10.1109/CVPR.2017.615http://dx.doi.org/10.1109/CVPR.2017.615］
Xu L， Lu C W， Xu Y and Jia J Y. 2011. Image smoothing via L0 gradient minimization. ACM Transactions on Graphics， 30（6）： 1-12 ［DOI： 10.1145/2070781.2024208http://dx.doi.org/10.1145/2070781.2024208］
Yang Y C and Soatto S. 2018. Conditional prior networks for optical flow//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 282-298 ［DOI： 10.1007/978-3-030-01267-0_17http://dx.doi.org/10.1007/978-3-030-01267-0_17］
You S S， Yao H T and Xu C S. 2021. Multi-target multi-camera tracking with optical-based pose association. IEEE Transactions on Circuits and Systems for Video Technology， 31（8）： 3105-3117 ［DOI： 10.1109/TCSVT.2020.3036467http://dx.doi.org/10.1109/TCSVT.2020.3036467］
Zhang C X， Chen Z， Wang M R and Li M. 2017. Non-local TV-L1 optical flow estimation using the weighted neighboring triangle filtering. Journal of Image and Graphics， 22（8）： 1056-1067
张聪炫， 陈震， 汪明润， 黎明. 2017. 非局部加权邻域三角滤波TV-L1光流估计. 中国图象图形学报， 22（8）： 1056-1067 ［DOI： 10.11834/jig.160594http://dx.doi.org/10.11834/jig.160594］
Zhang Z W， Yan J J， Liu S F， Lei Z， Yi D and Li S Z. 2012. A face antispoofing database with diverse attacks//Proceedings of the 5th IAPR International Conference on Biometrics. New Delhi， India： IEEE： 26-31 ［DOI： 10.1109/ICB.2012.6199754http://dx.doi.org/10.1109/ICB.2012.6199754］
Zimmer H， Bruhn A and Weickert J. 2011. Optic flow in harmony. International Journal of Computer Vision， 93（3）： 368-388 ［DOI： 10.1007/s11263-011-0422-6http://dx.doi.org/10.1007/s11263-011-0422-6］
相关文章
相关作者
相关机构
京公网安备11010802024621