Face detection and tracking algorithm for remote photoplethysmography

Changchen Zhao; Peiyi Mei; Yuanjing Feng

doi:10.11834/jig.200314

Physiological Signal and Psychological Analysis | Views : 0 下载量: 1597 CSCD: 0

PDF
Export
Share
Collection
Album

Face detection and tracking algorithm for remote photoplethysmography
Vol. 25, Issue 11, Pages: 2439-2450(2020)
Received：21 June 2020，

Revised：2020-8-11，

Accepted：18 August 2020，

Published：16 November 2020
DOI： 10.11834/jig.200314
稿件说明：

移动端阅览

DOI：

Changchen Zhao, Peiyi Mei, Yuanjing Feng. Face detection and tracking algorithm for remote photoplethysmography[J]. Journal of Image and Graphics, 2020, 25(11): 2439-2450. DOI： 10.11834/jig.200314.

摘要

目的

远程光体积描记（remote photoplethysmograph，rPPG）是一种基于视频的非接触心率测量方法，通过跟踪人脸皮肤区域并从中提取周期性微弱变化的颜色信号估计出心率。目前基于级联回归树的人脸地标方法训练的Dlib库，由于能快速准确定位人脸轮廓，正逐渐被研究者用于跟踪皮肤感兴趣区域（region of interest，ROI）。由于实际应用中存在地标无规则抖动，且现有研究没有考虑目标晃动的影响，因此颜色信号提取不准确，心率估计精度不佳。为了克服以上缺陷，提出一种基于Dlib的抗地标抖动和运动晃动的跟踪方法。

方法

本文方法主要包含两个步骤：首先，通过阈值判断两帧间地标的区别，若近似则沿用地标，反之使用当前帧地标以解决抖动问题。其次，针对运动晃动，通过左右眼地标中点计算旋转角度，矫正晃动的人脸，保证ROI在运动中也能保持一致。

结果

通过信噪比（signal-to-noise，SNR）、平均绝对误差（mean absolute error，MAE）和均方根误差（root mean squared error，RMSE）来评价跟踪方法在rPPG中的测量表现。经在UBFC-RPPG（stands for Univ.Bourgogne Franche-Comté Remote PhotoPlethysmoGraphy）和PURE（Pulse Rate Detection Dataset）数据集测试，与Dlib相比，本文方法rPPG测量结果在UBFC-RPPG中SNR提高了约0.425 dB，MAE提高0.291 5 bpm，RMSE降低0.645 3 bpm；在PURE中SNR降低了0.041 1 dB，MAE降低0.065 2 bpm，RMSE降低0.271 8 bpm。

结论

本文方法相比于Dlib有效提高跟踪框稳定性，在静止和运动中都能跟踪相同ROI，适合rPPG应用。

Abstract

Objective

Remote photoplethysmography (rPPG) is a video-based noncontact heart rate measurement method. It tracks the skin area of the face

extracts periodic subtle color variations within video data

and estimates heart rate from color signals. It has a broad application in the field of medical healthcare and daily living. Currently

facial landmark-based tracking methods are widely used by researchers to track regions of interest (ROIs) because it can quickly and accurately locate face contours. The Dlib library trained based on the cascade regression tree method is widely used. However

in practice

it has problems

such as the irregular jitter of landmarks during tracking

and present research does not consider the effect of target shaking. Thus

color signal extraction is inaccurate

and the accuracy of heart rate estimation is poor. To overcome the above problems

we first use the threshold method to stabilize landmarks

then rotate the image to correct the shaking face

and finally extract the region of interest and extract the color signal to estimate the heart rate.

Method

When Dlib is applied to a frame

it detects the face bounding box

fits a set of average landmark points in the model to the face frame as the first predicted landmark

and updates the landmark through a cascade of regression trees. In each regression tree

a tree node decides the direction of splitting on the basis of the difference in pixel intensity between two pixels in a graph and threshold

and the offset is obtained until the last layer. When the detected face position is different or the offset obtained by a certain tree is different

deviation appears between the landmarks of two frames

that is

the landmarks jitter irregularly. Dlib suffers from the problem of landmark jitter. In some low-head scenes

the degree of jitter is particularly large

but the contours of landmarks detected using two-frame images are approximate. Nevertheless

the facial landmark detection accuracy of Dlib is more accurate than most object detection and tracking algorithms. Accordingly

the proposed method for stabilizing landmarks is based on the threshold method. First

we employ Euclidean distance and the standard deviation of a landmark as the threshold

which can be used in determining the current movement state of a subject. A large standard deviation indicates that the difference between the two frames of the landmark is large. Conversely

a small standard deviation indicates that the difference between the two frames of landmarks is small

that is

the target may be stationary. In this paper

the landmark of a previous frame is used when a target is stationary or landmark jitter is strong. Otherwise

the landmarks are updated regularly. Second

for motion shake

for maintaining the straight position of faces in images

a rotation correction mechanism is proposed. It calculates the rotation angle through the midpoint of the left and right eye landmarks for image rotation

then maps landmarks to the rotated image

and finally extracts an ROI to ensure that the ROI is consistent.

Result

This paper evaluates the performance of a tracking method for rPPG pulse extraction by using signal-to-noise ratio (SNR). SNR represents the quality of the pulse signal estimated from the color signals extracted from the tracking area. This paper selects UBFC-RPPG(stands for Univ. Bourgogne Franche-Comté Remote PhotoPlethysmoGraphy) and PURE(Pulse Rate Detection Dataset) datasets to test method performance. Compared with Dlib

the proposed method improves SNR by 0.425 dB and root mean squared error(RMSE) decreases by 0.645 3 bpm but mean absolute eror(MAE) increases by 0.291 5 bpm on UBFC-RPPG dataset. MAE decreases by 0.065 2 bpm and RMSE decreases by 0.271 8 bpm but SNR decreases by 0.0411 dB on PURE dataset.

Conclusion

Compared with Dlib

the proposed method effectively improves the stability of the tracking frame and can track the same ROI still and moving images of the subject in still or moving situation. It is a tracking method suitable for rPPG applications.

关键词

Keywords

references

Asthana A, Zafeiriou S, Cheng S Y and Pantic M. 2013. Robust discriminative response map fitting with constrained local models//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE: 3444-3451[ DOI: 10.1109/CVPR.2013.442 http://dx.doi.org/10.1109/CVPR.2013.442 ]

Bertinetto L, Valmadre J, Golodetz S, Miksik O and Torr P H S. 2016. Staple: complementary learners for real-time tracking//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 1401-1409[ DOI: 10.1109/CVPR.2016.156 http://dx.doi.org/10.1109/CVPR.2016.156 ]

Bobbia S, Macwan S, Benezeth Y, Mansouri A and Dubois J. 2019. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters, 124:82-90[DOI:10.1016/j.patrec.2017.10.017]

Casalino G, Castellano G, Pasquadibisceglie V and Zaza G. 2019. Contact-less real-time monitoring of cardiovascular risk using video imaging and fuzzy inference rules. Information, 10(1):#9[DOI:10.3390/info10010009]

de Haan G and Jeanne V. 2013. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 60(10):2878-2886[DOI:10.1109/TBME.2013.2266196]

Henriques J F, Caseiro R, Martins P and Batista J. 2012. Exploiting the circulant structure of tracking-by-detection with kernels//Proceedings of the 12th European Conference on Computer Vision Florence. Italy: Springer: 702-715[ DOI: 10.1007/978-3-642-33765-9_50 http://dx.doi.org/10.1007/978-3-642-33765-9_50 ]

Kazemi V and Sullivan J. 2014. One millisecond face alignment with an ensemble of regression trees//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE: 1867-1874[ DOI: 10.1109/CVPR.2014.241 http://dx.doi.org/10.1109/CVPR.2014.241 ]

Klaessens J H G M, van den Born M, van der Veen A, Sikkens-van de Kraats J, van den Dungen F A M and Verdaasdonk R M. 2014. Development of a baby friendly non-contact method for measuring vital signs: first results of clinical measurements in an open incubator at a neonatal intensive care unit//Proceedings of SPIE 8935, Advanced Biomedical and Clinical Diagnostic Systems XII. San Francisco: SPIE: #8935[ DOI: 10.1117/12.2038353 http://dx.doi.org/10.1117/12.2038353 ]

Lee K, Han D K and Ko H. 2018. Video analytic based health monitoring for driver in moving vehicle by extracting effective heart rate inducing features. Journal of Advanced Transportation, 2018: #8513487[ DOI: 10.1155/2018/8513487 http://dx.doi.org/10.1155/2018/8513487 ]

Lin Y C and Lin Y H. 2018. Step count and pulse rate detection based on the contactless image measurement method. IEEE Transactions on Multimedia, 20(8):2223-2231[DOI:10.1109/TMM.2018.2790172]

Liu Y, Ouyang J F and Yan Y G. 2016. Method of measuring heart rate using a Webcam. Computer Engineering and Applications, 52(7):210-214

刘祎, 欧阳健飞, 闫勇刚. 2016.基于普通摄像头的心率测量方法研究.计算机工程与应用, 52(7):210-214 [DOI:10.3778/j.issn.1002-8331.1404-0183]

Ma B and Wang R Y. 2006. Resting heart rate and cardiovascular diseases. Advances in Cardiovascular Diseases, 27(S1):90-92

马斌, 王瑞英. 2006.静息心率与心血管疾病的研究进展.心血管病学进展, 27(S1):90-92 [DOI:10.3969/j.issn.1004-3934.2006.z1.029]

Qi G, Yang X Z, Wu X and Huo L. 2017. Heart rate detection for non-cooperative shaking face. Journal of Image and Graphics, 22(1):126-136

戚刚, 杨学志, 吴秀, 霍亮. 2017.非合作面部晃动情况下的心率检测.中国图象图形学报, 22(1):126-136 [DOI:10.11834/jig.20170114]

Shi J B and Tomasi. 1994. Good features to track//Proceedings of 1994 IEEE Conference on Computer Vision and Pattern Recognition. Seattle: IEEE: 593-600[ DOI: 10.1109/CVPR.1994.323794 http://dx.doi.org/10.1109/CVPR.1994.323794 ]

Stricker R, Müller S and Gross H M. 2014. Non-contact video-based pulse rate measurement on a mobile service robot//Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication. Edinburgh: 1056-1062[ DOI: 10.1109/ROMAN.2014.6926392 http://dx.doi.org/10.1109/ROMAN.2014.6926392 ]

Van Gastel M, Stuijk S and De Haan G. 2015. Motion robust remote-PPG in infrared. IEEE Transactions on Biomedical Engineering, 62(5):1425-1433[DOI:10.1109/TBME.2015.2390261]

Wang W J, den Brinker A C, Stuijk S and de Haan G. 2017a. Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7):1479-1491[DOI:10.1109/TBME.2016.2609282]

Wang W J, den Brinker A C, Stuijk S and de Haan G. 2017b. Robust heart rate from fitness videos. Physiological Measurement, 38(6):1023-1044[DOI:10.1088/1361-6579/aa6 d02]

Zhang Q, Wu Q T, Zhou Y M, Wu X Y, Ou Y and Zhou H Z. 2017. Webcam-based, non-contact, real-time measurement for the physiological parameters of drivers. Measurement, 100:311-321[DOI:10.1016/j.measurement.2017.01.007]

Zhao C C, Mei P Y, Xu S S, Li Y Q and Feng Y J. 2019. Performance evaluation of visual object detection and tracking algorithms used in remote photoplethysmography//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul: IEEE: 1646-1655[ DOI: 10.1109/ICCVW.2019.00204 http://dx.doi.org/10.1109/ICCVW.2019.00204 ]

Zhao Y T, Fu M L, Wang B, Zhang X G and Li X L. 2016. Heart rate measurements using principal component analysis and overlapping histgram based on video signal. Chinese Journal of Biomedical Engineering, 35(3):284-291

赵彦涛, 付美玲, 王斌, 张旭光, 李小俚. 2016.基于主成分分析和重叠直方图统计的视频信号心率测量.中国生物医学工程学报, 35(3):284-291 [DOI:10.3969/j.issn.0258-8021.2016.03.005]