浏览全部资源
扫码关注微信
Received：01 July 2020，
Revised：2020-8-14，
Accepted：21 August 2020，
Published：16 November 2020
移动端阅览
远程光电容积脉搏波描记法(remote photoplethysmography，rPPG)是指通过摄像头等传感器来捕捉由心动周期造成的皮肤颜色周期性变化的技术。利用rPPG技术可以提取血液体积脉冲信号，进而测量心率、呼吸率和心跳变异性等心动周期相关的生理指标。近年基于rPPG的生理指标测量方法取得了飞速发展，准确性和鲁棒性已得到了大幅提高。该类技术的算法流程主要包括图像/视频的获取和感兴趣区域提取、血液体积脉冲信号提取和生理指标测量等步骤。基于这一算法流程，本文从算法所依据的假设或先验知识出发，对相关文献进行了总结和讨论。此外，还从评测任务、评测数据、评测指标及评测协议等4个方面系统整理针对基于rPPG的生理指标测量方法的评价体系。最后，本文讨论了该领域当前所面临的挑战并展望了可能的技术路线。
Physiological signals
such as heart rate (HR)
respiration frequency (RF)
and heart rate variability (HRV)
are important clues to analyze a person's health and affective status. Traditional measurements of physiological signals are based on the electrocardiography (ECG) or contact photoplethysmography (cPPG) technology. However
both technologies require professional equipment
which may cause inconvenience and discomfort for subjects. Remote photoplethysmography (rPPG) technology for remote measurement of physiological signals has progressed considerably and recently attracted considerable research attention. The rPPG technology
which is based on skin color variations due to the periodical optical absorption of skin tissue caused by cardiac activity
demonstrates high potential in many applications
such as healthcare
sleep monitoring
and defection detection. The process for rPPG-based physiological measurement can be divided into three steps. First
regions of interest (ROIs) are extracted from the face video. Second
blood volume pulse(BVP) signal is reconstructed from signals generated from the ROIs. Finally
the reconstructed BVP signal is used for physiological measurements. The reconstruction of the BVP signal is the key step for rPPG-based remote physiological measurements. A detailed review of methods for rPPG-based remote physiological measurement is presented in this study from the aspect of assumptions they use
which can be categorized into three kinds
i.e.
methods based on the skin reflection model
methods based on the BVP signal's physical characteristics
and data-driven methods. Studies on the skin reflection model-based methods can be further categorized into spatial skin and skin reflection models of different color channels. Studies on methods that using the BVP signal's physical characteristics can be further categorized into blind signal separation
manifold projection
low rank factorization
and frequency domain constraint. Studies on data-driven methods can be further categorized into methods based on hand-crafted features and deep learning. A detailed review of evaluations of different rPPG-based physiological measurement methods is also presented from the aspects of tasks
databases
metrics
and protocols. Evaluation tasks used for remote physiological measurement include average heart rate measurement
respiration frequency measurement
and heart rate variability analysis. Databases of rPPG-based physiological measurements are summarized according to database scale and variations. Evaluation metrics for remote physiological measurement can be categorized into statistics of error
correlation
and signal quality. Evaluation protocols for data-driven methods are summarized into fixed partition
subject-independent division
subject-exclusive division
and cross-database protocols. Finally
we discuss the challenges of the rPPG-based remote physiological measurement and put forward the potential research directions for future investigations. Challenges include video quality (i.e.
video compression and pre-processing of frames)
influence of subject's head movements
variations of lighting conditions
and lacking data. Future research trends include designing hand-crafted methods for different challenge scenarios and exploring technologies
such as self-supervised
semi-supervised
and weakly-supervised learning
for data-driven methods.
Akselrod S, Gordon D, Ubel F A, Shannon D C, Berger A C and Cohen R J. 1981. Power spectrum analysis of heart rate fluctuation:a quantitative probe of beat-to-beat cardiovascular control. Science, 213(4504):220-222[DOI:10.1126/science.6166045]
Al-Naji A, Gibson K, Lee S H and Chahl J. 2017. Monitoring of cardiorespiratory signal:principles of remote measurements and review of methods. IEEE Access, 5:15776-15790[DOI:10.1109/ACCESS.2017.2735419]
Balakrishnan G, Durand F and Guttag J. 2013. Detecting pulse from head motions in video//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE: 3430-3437[ DOI: 10.1109/CVPR.2013.440 http://dx.doi.org/10.1109/CVPR.2013.440 ]
Bobbia S, Macwan R, Benezeth Y, Mansouri A and Dubois J. 2019. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters, 124(6):82-90[DOI:10.1016/j.patrec.2017.10.017]
Bousefsaf F, Maaoui C and Pruski A. 2013. Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomedical Signal Processing and Control, 8(6):568-574[DOI:10.1016/j.bspc.2013.05.010]
Chen W X and McDuff D. 2018. DeepPhys: video-based physiological measurement using convolutional attention networks//Proceedings of the 15th European Conference on Computer Vision. Munich: Springer: 349-365[ DOI: 10.1007/978-3-030-01216-8_22 http://dx.doi.org/10.1007/978-3-030-01216-8_22 ]
Chen X, Cheng J, Song R C, Liu Y, Ward R and Wang Z J. 2019. Video-based heart rate measurement:recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement, 68(10):3600-3615[DOI:10.1109/TIM.2018.2879706]
de Haan G and Jeanne V. 2013. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 60(10):2878-2886[DOI:10.1109/TBME.2013.2266196]
de Haan G and Van Leest A. 2014. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiological Measurement, 35(9):1913-1926[DOI:10.1088/0967-3334/35/9/1913]
Feng L T, Po L M, Xu X Y, Li Y M, Cheung C H, Cheung K W and Yuan F. 2015a. Dynamic ROI based on K-means for remote photoplethysmography//Proceedings 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing. Brisbane: IEEE: 1310-1314[ DOI: 10.1109/ICASSP.2015.7178182 http://dx.doi.org/10.1109/ICASSP.2015.7178182 ]
Feng T, Po L M, Xu X Y, Li Y M and Ma R Y. 2015b. Motion-resistant remote imaging photoplethysmography based on the optical properties of skin. IEEE Transactions on Circuits and Systems for Video Technology, 25(5):879-891[DOI:10.1109/TCSVT.2014.2364415]
Garbey M, Sun N F, Merla A and Pavlidis I. 2007. Contact-free measurement of cardiac pulse based on the analysis of thermal imagery. IEEE Transactions on Biomedical Engineering, 54(8):1418-1426[DOI:10.1109/TBME.2007.891930]
Gudi A, Bittner M, Lochmans R and Van Gemert J. 2019. Efficient real-time camera based estimation of heart rate and its variability//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul: IEEE: #00196[ DOI: 10.1109/ICCVW.2019.00196 http://dx.doi.org/10.1109/ICCVW.2019.00196 ]
Hassan M A, Malik A S, Fofi D, Saad N, Karasfi B, Ali Y S and Meriaudeau F. 2017. Heart rate estimation using facial video:a review. Biomedical Signal Processing and Control, 38:346-360[DOI:10.1016/j.bspc.2017.07.004]
Heusch G, Anjos A and Marcel S. 2017. A reproducible study on remote heart rate measurement[EB/OL].[2020-06-01] . https://arxiv.org/pdf/1709.00962.pdf https://arxiv.org/pdf/1709.00962.pdf
Hsu G S, Ambikapathi A and Chen M S. 2017. Deep learning with time-frequency representation for pulse estimation from facial videos//Proceedings of 2017 IEEE International Joint Conference on Biometrics. Denver: IEEE: 383-389[ DOI: 10.1109/BTAS.2017.8272721 http://dx.doi.org/10.1109/BTAS.2017.8272721 ]
Hsu Y C, Lin Y L and Hsu W. 2014. Learning-based heart rate detection from remote photoplethysmography features//Proceedings of 2014 IEEE International Conference on Acoustics, Speech, and Signal Processing. Florence: IEEE: 4433-4437[ DOI: 10.1109/ICASSP.2014.6854440 http://dx.doi.org/10.1109/ICASSP.2014.6854440 ]
Huang J F, Bai G C, Xiong N X and Wei J G. 2020. Non-touch heart rate estimation based on the low-rank and sparse matrix decomposition. Journal of Graphics, 41(1):66-72
黄继风, 白国臣, 熊乃学, 魏建国. 2020.基于低秩稀疏矩阵分解的非接触心率估计.图学学报, 41(1):66-72[DOI:10.11996/JG.j.2095-302X.2020010066]
Khanam F T Z, Al-Naji A and Chahl J. 2019. Remote monitoring of vital signs in diverse non-clinical and clinical scenarios using computer vision systems:a review. Applied Sciences, 9(20):#4474[DOI:10.3390/app9204474]
Kranjec J, Beguš S, Geršak G and Drnovšek J. 2014. Non-contact heart rate and heart rate variability measurements:a review. Biomedical Signal Processing and Control, 13:102-112[DOI:10.1016/j.bspc.2014.03.004]
Kumar M, Veeraraghavan A and Sabharwal A. 2015. DistancePPG:robust non-contact vital signs monitoring using a camera. Biomedical Optics Express, 6(5):1565-1588[DOI:10.1364/BOE.6.001565]
Lam A and Kuno Y. 2015. Robust heart rate measurement from video using select random patches//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago: IEEE: 3640-3648[ DOI: 10.1109/ICCV.2015.415 http://dx.doi.org/10.1109/ICCV.2015.415 ]
Lewandowska M, Rumiński J, Kocejko T and Nowak J. 2011. Measuring pulse rate with a webcam-a non-contact method for evaluating cardiac activity//Proceedings of 2011 Federated Conference on Computer Science and Information Systems. Szczecin: IEEE: 405-410
Li C Z, Cummings J, Lam J, Graves E and Wu W. 2009. Radar remote monitoring of vital signs. IEEE Microwave Magazine, 10(1):47-56[DOI:10.1109/MMM.2008.930675]
Li X B, Alikhani I, Shi J G, Seppanen T, Junttila J, Majamaa-Voltti K, Tulppo M and Zhao G Y. 2018. The OBF database: a large face video database for remote physiological signal measurement and atrial fibrillation detection//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Xi'an: IEEE: 242-249[ DOI: 10.1109/FG.2018.00043 http://dx.doi.org/10.1109/FG.2018.00043 ]
Li X B, Chen J, Zhao G Y and Pietikäinen M. 2014. Remote heart rate measurement from face videos under realistic situations//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE: 4264-4271[ DOI: 10.1109/CVPR.2014.543 http://dx.doi.org/10.1109/CVPR.2014.543 ]
Manzone T A, Dam H Q, Soltis D and Sagar V V. 2007. Blood volume analysis:a new technique and new clinical interest reinvigorate a classic study. Journal of Nuclear Medicine Technology, 35(2):55-63[DOI:10.2967/jnmt.106.035972]
McDuff D, Gontarek S and Picard R W. 2014. Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Transactions on Biomedical Engineering, 61(10):2593-2601[DOI:10.1109/TBME.2014.2323695]
McDuff D J, Blackford E B and Estepp J R. 2017. The impact of video compression on remote cardiac pulse measurement using imaging photoplethysmography//Proceedings of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Washington: IEEE: 63-70[ DOI: 10.1109/FG.2017.17 http://dx.doi.org/10.1109/FG.2017.17 ]
McDuff D J, Estepp J R, Piasecki A M and Blackford E B. 2015. A survey of remote optical photoplethysmographic imaging methods//Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Milan: IEEE: 6398-6404[ DOI: 10.1109/EMBC.2015.7319857 http://dx.doi.org/10.1109/EMBC.2015.7319857 ]
Mesleh A, Skopin D, Baglikov S and Quteishat A. 2012. Heart rate extraction from vowel speech signals. Journal of Computer Science and Technology, 27(6):1243-1251[DOI:10.1007/s11390-012-1300-6]
Niu X S, Han H, Shan S G and Chen X L. 2017. Continuous heart rate measurement from face: a robust rPPG approach with distribution learning//Proceedings of 2017 IEEE International Joint Conference on Biometrics. Denver: IEEE: 642-650[ DOI: 10.1109/BTAS.2017.8272752 http://dx.doi.org/10.1109/BTAS.2017.8272752 ]
Niu X S, Han H, Shan S G and Chen X L. 2018a. SynRhythm: learning a deep heart rate estimator from general to specific//Proceedings of the 24th International Conference on Pattern Recognition. Beijing: IEEE: 3580-3585[ DOI: 10.1109/ICPR.2018.8546321 http://dx.doi.org/10.1109/ICPR.2018.8546321 ]
Niu X S, Han H, Shan S G and Chen X L. 2018b. VIPL-HR: a multi-modal database for pulse estimation from less-constrained face video//Proceedings of the 14th Asian Conference on Computer Vision. Australia: Springer: 1-16[ DOI: 10.1007/978-3-030-20873-8_36 http://dx.doi.org/10.1007/978-3-030-20873-8_36 ]
Niu X S, Zhao X Y, Han H, Das A, Dantcheva A, Shan S G and Chen X L. 2019. Robust remote heart rate estimation from face utilizing spatial-temporal attention//Proceedings of the 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE: #8756554[ DOI: 10.1109/FG.2019.8756554 http://dx.doi.org/10.1109/FG.2019.8756554 ]
Niu X S, Shan S G, Han H and Chen X L. 2020. RhythmNet:end-to-end heart rate estimation from face via spatial-temporal representation. IEEE Transactions on Image Processing, 29:2409-2423[DOI:10.1109/TIP.2019.2947204]
Noble D. 1984. The surprising heart:a review of recent progress in cardiac electrophysiology. The Journal of Physiology, 353(1):1-50[DOI:10.1113/jphysiol.1984.sp015320]
Nowara E and McDuff D. 2019. Combating the impact of video compression on non-contact vital sign measurement using supervised learning//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul: IEEE: #211[ DOI: 10.1109/ICCVW.2019.00211 http://dx.doi.org/10.1109/ICCVW.2019.00211 ]
Osman A, Turcot J and El Kaliouby R. 2015. Supervised learning approach to remote heart rate estimation from facial videos//Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Ljubljana: IEEE: #7163150[ DOI: 10.1109/FG.2015.7163150 http://dx.doi.org/10.1109/FG.2015.7163150 ]
Pilz C S, Zaunseder S, Krajewski J and Blazek V. 2018. Local group invariance for heart rate estimation from face videos in the wild//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE: 1335-13358[ DOI: 10.1109/CVPRW.2018.00172 http://dx.doi.org/10.1109/CVPRW.2018.00172 ]
Pilz C S. 2019. On the vector space in photoplethysmography imaging//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul: IEEE: #197[ DOI: 10.1109/ICCVW.2019.00197 http://dx.doi.org/10.1109/ICCVW.2019.00197 ]
Po L M, Feng L T, Li Y M, Xu X Y, Cheung T C H and Cheung K. W. 2018. Block-based adaptive ROI for remote photoplethysmography. Multimedia Tools and Applications, 77(6):6503-6529[DOI:10.1007/s11042-017-4563-7]
Poh M Z, McDuff J D and Picard W R. 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10):10762-10774[DOI:10.1364/OE.18.010762]
Poh M Z, McDuff J D and Picard W R. 2011. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1):7-11[DOI:10.1109/TBME.2010.2086456]
Qiu Y, Liu Y, Arteaga-Falconi J, Dong H W and El Saddik A. 2019. EVM-CNN:real-time contactless heart rate estimation from facial video. IEEE Transactions on Multimedia, 21(7):1778-1787[DOI:10.1109/TMM.2018.2883866]
Soleymani M, Lichtenauer J, Pun T and Pantic M. 2012. A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing, 3(1):42-55[DOI:10.1109/T-AFFC.2011.25]
Spetlik R, Franc V, Cech J and Matas J. 2018. Visual heart rate estimation with convolutional neural network//Proceedings of British Machine Vision Conference. Newcastle: BMVA Press: #329044207
Stricker R, Müller S and Gross H M. 2014. Non-contact video-based pulse rate measurement on a mobile service robot//Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication. Edinburgh: IEEE: 1056-1062[ DOI: 10.1109/ROMAN.2014.6926392 http://dx.doi.org/10.1109/ROMAN.2014.6926392 ]
Sun Y and Thakor N. 2016. Photoplethysmography revisited:from contact to noncontact, from point to imaging. IEEE Transactions on Biomedical Engineering, 63(3):463-477[DOI:10.1109/TBME.2015.2476337]
Tarassenko L, Villarroel M, Guazzi A, Jorge J, Clifton D A and Pugh C. 2014. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiological Measurement, 35(5):807-831[DOI:10.1088/0967-3334/35/5/807]
Tasli H E, Gudi A and Den Uyl M. 2014. Remote PPG based vital sign measurement using adaptive facial regions//Proceedings of 2014 IEEE International Conference on Image Processing. Paris: IEEE: 1410-1414[ DOI: 10.1109/ICIP.2014.7025282 http://dx.doi.org/10.1109/ICIP.2014.7025282 ]
Teichmann D, Brüser C, Eilebrecht B, Abbas A, Blanik N and Leonhardt S. 2012. Non-contact monitoring techniques-principles and applications//Proceedings of 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. San Diego: IEEE: 1302-1305[ DOI: 10.1109/EMBC.2012.6346176 http://dx.doi.org/10.1109/EMBC.2012.6346176 ]
Tsouri G R, Kyal S, Dianat S A and Mestha L K. 2012. Constrained independent component analysis approach to nonobtrusive pulse rate measurements. Journal of Biomedical Optics, 17(7):077011[DOI:10.1117/1.JBO.17.7.077011]
Tulyakov S, Alameda-Pineda X, Ricci E, Yin L J, Cohn J F and Sebe N. 2016. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 2396-2404[ DOI: 10.1109/CVPR.2016.263 http://dx.doi.org/10.1109/CVPR.2016.263 ]
Van Gastel M, Stuijk S and De Haan G. 2015. Motion robust remote-PPG in infrared. IEEE Transactions on Biomedical Engineering, 62(5):1425-1433[DOI:10.1109/TBME.2015.2390261]
Verkruysse W, Svaasand L O and Nelson J S. 2008. Remote plethysmographic imaging using ambient light. Optics Express, 16(26):21434-21445[DOI:10.1364/oe.16.021434]
Wang W J, Stuijk S and De Haan G. 2015. Exploiting spatial redundancy of image sensor for motion robust rPPG. IEEE Transactions on Biomedical Engineering, 62(2):415-425[DOI:10.1109/TBME.2014.2356291]
Wang W J, Den Brinker A C, Stuijk S and De Haan G. 2017a. Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7):1479-1491[DOI:10.1109/TBME.2016.2609282]
Wang W J, Den Brinker A C, Stuijk S and De Haan G. 2017b. Color-distortion filtering for remote photoplethysmography//Proceedings of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Washington: IEEE: 71-78[ DOI: 10.1109/FG.2017.18 http://dx.doi.org/10.1109/FG.2017.18 ]
Wang W J, Den Brinker A C, Stuijk S and De Haan G. 2017c. Amplitude-selective filtering for remote-PPG. Biomedical Optics Express, 8(3):1965-1980[DOI:10.1364/BOE.8.001965]
Wei L, Tian Y H, Wang Y W, Ebrahimi T and Huang T J. 2012. Automatic webcam-based human heart rate measurements using Laplacian eigenmap//Proceedings of the 11th Asian Conference on Computer Vision. Daejeon: Springer: 281-292[ DOI: 10.1007/978-3-642-37444-9_22 http://dx.doi.org/10.1007/978-3-642-37444-9_22 ]
Wu H Y, Rubinstein M, Shih E, Guttag J, Durand F and Freeman W. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics, 31(4):#65[DOI:10.1145/2185520.2185561]
Yasuma F and Hayano J I. 2004. Respiratory sinus arrhythmia:why does the heartbeat synchronize with respiratory rhythm? Chest, 125(2):683-690[DOI:10.1378/chest.125.2.683]
Yu Z T, Li X B and Zhao G Y. 2019a. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks//Proceedings of the 30th British Machine Vision Conference. Cardiff: BMVA Press
Yu Z T, Peng W, Li X B, Hong X P and Zhao G Y. 2019b. Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE: 151-160[ DOI: 10.1109/ICCV.2019.00024 http://dx.doi.org/10.1109/ICCV.2019.00024 ]
Zhou S, Yang X Z, Jin J, Fang S and Liu X N. 2019. Non-contact heart rate detection using self-adaptive signal recovery algorithm. Journal of Image and Graphics, 24(10):1670-1682
周双, 杨学志, 金兢, 方帅, 刘雪南. 2019.采用自适应信号恢复算法的非接触式心率检测.中国图象图形学报, 24(10):1670-1682[DOI:10.11834/jig.190013]
相关作者
相关机构
京公网安备11010802024621
