Fatigue driving detection based on ocular self-quotient image and gradient image co-occurrence matrix

Jiankai Pan; Zhengqing Liu; Qiucheng Wang

doi:10.11834/jig.200258

Object Detection and Tracking | Views : 0 下载量: 173 CSCD: 2

PDF
Export
Share
Collection
Album

Fatigue driving detection based on ocular self-quotient image and gradient image co-occurrence matrix
Vol. 26, Issue 1, Pages: 154-164(2021)
Received：03 June 2020，

Revised：2020-10-13，

Accepted：20 October 2020，

Published：16 January 2021
DOI： 10.11834/jig.200258
稿件说明：

移动端阅览

DOI：

Jiankai Pan, Zhengqing Liu, Qiucheng Wang. Fatigue driving detection based on ocular self-quotient image and gradient image co-occurrence matrix[J]. Journal of Image and Graphics, 2021, 26(1): 154-164. DOI： 10.11834/jig.200258.

摘要

目的

疲劳驾驶是引发车辆交通事故的主要原因之一，针对现有方法在驾驶员面部遮挡情况下对眼睛状态识别效果不佳的问题，提出了一种基于自商图—梯度图共生矩阵的驾驶员眼部疲劳检测方法。

方法

利用以残差网络（residual network，ResNet）为前置网络的SSD（single shot multibox detector）人脸检测器来获取视频中的有效人脸区域，并通过人脸关键点检测算法分割出眼睛局部区域图像；建立驾驶员眼部的自商图与梯度图共生矩阵模型，分析共生矩阵的数字统计特征，选取效果较好的特征用以判定人眼的开闭状态；结合眼睛闭合时间百分比（percentage of eyelid closure，PERCLOS）与最长闭眼持续时间（maximum closing duration，MCD）两个疲劳指标来判别驾驶员的疲劳状态。

结果

在六自由度汽车性能虚拟仿真实验平台上模拟汽车驾驶，采集并分析驾驶员面部视频，本文方法能够有效识别驾驶员面部遮挡时眼睛的开闭状态，准确率高达99.12%，面部未遮挡时的识别精度为98.73%，算法处理视频的速度约为32帧/s。对比方法1采用方向梯度直方图特征与支持向量机分类器相结合的人脸检测算法，并以眼睛纵横比判定开闭眼状态，在面部遮挡时识别较弱；以卷积神经网络（convolutional neural network，CNN）判别眼睛状态的对比方法2虽然在面部遮挡情况下的准确率高达98.02%，但眨眼检测准确率效果不佳。

结论

基于自商图—梯度图共生矩阵的疲劳检测方法能够有效识别面部遮挡时眼睛的开闭情况和驾驶员的疲劳状态，具有较快的检测速度与较高的准确率。

Abstract

Objective

Driver fatigue is known to be directly related to road safety and is a leading cause of traffic fatalities and injuries of seated drivers. Previous studies used many fatigue driving detection methods to detect and analyze the fatigue status of seated drivers. These methods aim to improve detection accuracy and usually include driving behavioral features (e.g.

steering wheel motion

lane keeping) and physiological features (e.g.

eye and face movement

heart rate variability

electroencephalogram

electroocoulogram

electrocardiogram). Physiological features

such as eye movement

are widely used to predict driver fatigue because they are nonintrusive and independent on the driving context. However

fatigue driving detection under occluded face conditions is challenging and needs a robust algorithm of eye feature extraction. The literature showed that most eye tracking methods require high-resolution images. This condition leads to low processing speed and difficultly on real-time eye tracking. In this study

a fatigue driving detection method based on self-quotient image (SQI) and gradient image co-occurrence matrix was presented. This improved method is based on gray level and gradient co-occurrence matrix. The proposed method provides a new approach for predicting fatigue status on driver fatigue applications in a short time.

Method

In this study

a six-degree of freedom vibration table and driving simulator were used to model the driving context. The eye fatigued state of seated driver in real time was recorded by using an RGB camera mounted in the front of the driver. A single shot multibox detector face detection algorithm was used to extract the driver's facial region from the recorded video with ResNet10 as the front network. An ensemble of regression trees facial landmark location algorithm was used to calibrate the driver's eye area for each frame of the recorded video. A gray-level image and an SQI of each frame was combined to obtain the co-occurrence matrix of the driver's eye image. The statistic of numerical characteristics of the SQI and gradient image co-occurrence matrix of driver's eye images was analyzed. The driver's eye states were determined in accordance with the statistical features of the matrix and had a large variation. Percentage of eyelid closure(PERCLOS) over the pupil over time and maximum closing duration(MCD) were utilized to predict the state transition of seated driver from nonfatigue to fatigue.

Result

We compared our method with other fatigue driving detection methods based on eye aspect ratio and based on convolutional neural network (CNN) model. The quantitative evaluation metrics contained face detection accuracy

detection accuracy of open and closed states of the eye

and blink detection accuracy. The experiment results show that our model has better performance than other methods in video datasets. These datasets were captured in the driving context modeled by six-degree of freedom vibration table and driving simulator. The following cases were studied to predict the fatigue status of seated driver. One case was detected under occluded face conditions (e.g.

wearing a mask and wearing glass)

and the other case had no mask or glass. The accuracy of eye state analysis of method 1 is 97.68%

but the accuracy depends on the positioning accuracy of facial landmarks. However

the overall detection speed of the algorithm is slow and sensitive to whether the driver wore a mask. This condition makes the detection of the eye state module invalid. Classifying the driver eye state with the CNN takes a long time

and the eye state accuracy of method 2 is more than 96%. However

the success rate of blink detection is approximately zero for hard video samples. As a comparison

the proposed algorithm has better performance in face detection

eye opening and closing

and blink accuracies. The accuracy of opening and closing the eyes reaches 98.73% for a video sample of drivers' face under occlusion

such as wearing a mask. The recognition precision is 99.52% for a video sample of drivers without mask or glass

and the video sample processing frame rate is up to 32 frame/s.

Conclusion

In this study

we proposed a fatigue driving detection method based on SQI and gradient image co-occurrence matrix. The experiment results demonstrate that our method has better performance than other several fatigue driving detection methods and can detect effectively the driver's eyes state when they open and close in real time. The proposed method has high accuracy and processing speed.

关键词

Keywords

references

Dai S Q and Zeng Z Y. 2018. Fatigue driving detection algorithm based on deep learning. Computer Systems and Applications, 27(7):113-120

戴诗琪, 曾智勇. 2018.基于深度学习的疲劳驾驶检测算法.计算机系统应用, 27(7):113-120[DOI:10.15888/j.cnki.csa.006415]

Dinges D F and Grace R. 1998. PERCLOS:A Valid Psychophysiological Measure of Alertness As Assessed by Psychomotor Vigilance. FHWA-MCRT-98-006. Federal Highway Administration

Dong Z X, Sun S Q, Wu Q and Xu J F. 2010. Study of correlation between heart rate variability and driving fatigue. Journal of Zhejiang University (Engineering Science), 44(1):46-50

董占勋, 孙守迁, 吴群, 徐娟芳. 2010.心率变异性与驾驶疲劳相关性研究.浙江大学学报(工学版), 44(1):46-50[DOI:10.3785/j.issn.1008-973X.2010.01.009]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 770-778[ DOI:10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Hong J G. 1984. Gray level-gradient co-occurrence matrix texture analysis method. Acta Automatica Sinica, 10(1):22-25

洪继光. 1984.灰度-梯度共生矩阵纹理分析方法.自动化学报, 10(1):22-25[DOI:10.16383/j.aas.1984.01.005]

Kazemi V and Sullivan J. 2014. One millisecond face alignment with an ensemble of regression trees//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 1867-1874[ DOI:10.1109/CVPR.2014.241 http://dx.doi.org/10.1109/CVPR.2014.241 ]

Kuang W T, Mao K C, Huang J C and Li H B. 2016. Fatigue driving detection based on sclera Gaussian model. Journal of Image and Graphics, 21(11):1515-1522

旷文腾, 毛宽诚, 黄家才, 李海彬. 2016.基于高斯眼白模型的疲劳驾驶检测.中国图象图形学报, 21(11):1515-1522[DOI:10.11834/jig.20161111]

Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y and Berg A C. 2016. SSD: single shot MultiBox detector//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 21-37[ DOI:10.1007/978-3-319-46448-0_2 http://dx.doi.org/10.1007/978-3-319-46448-0_2 ]

Niu Q N, Zhou Z Q, Jin L S, Liu W C and Yu P C. 2015. Detection of driver fatigue based on eye movements. Journal of Harbin Engineering University, 36(3):394-398

牛清宁, 周志强, 金立生, 刘文超, 于鹏程. 2015.基于眼动特征的疲劳驾驶检测方法.哈尔滨工程大学学报, 36(3):394-398[DOI:10.3969/j.issn.1006-7043.201311044]

Pan Z G, Liu R F and Zhang M M. 2019. Fatigue driving detection algorithm based on fuzzy comprehensive evaluation. Journal of Software, 30(10):2954-2963

潘志庚, 刘荣飞, 张明敏. 2019.基于模糊综合评价的疲劳驾驶检测算法研究.软件学报, 30(10):2954-2963[DOI:10.13328/j.cnki.jos.005779]

Song F Y, Tan X Y, Liu X and Chen S C. 2014. Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recognition, 47(9):2825-2838[DOI:10.1016/j.patcog.2014.03.024]

Song K, Liu N, Nie X and Yang D. 2013. Recognition of eye opening and closing state based on Gabor transform. Journal of Jilin University (Information Science Edition), 31(6):555-561

宋凯, 刘娜, 聂鑫, 杨东. 2013.基于Gabor变换的人眼开闭状态识别技术研究.吉林大学学报(信息科学版), 31(6):555-561[DOI:10.3969/j.issn.1671-5896.2013.06.001]

Soukupova T and Cech J. 2016. Real-time eye blink detection using facial landmarks//Proceedings of the 21st computer vision winter workshop[EB/OL ] .[2020-05-13 ] . http://cmp.felk.cvut.cz/ftp/articles/cech/Soukupova-CVWW-2016.pdf http://cmp.felk.cvut.cz/ftp/articles/cech/Soukupova-CVWW-2016.pdf

Viola P and Jones M. 2001. Rapid object detection using a boosted cascade of simple features//Proceedings of 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Kauai, HI, USA: IEEE: 511-518[ DOI:10.1109/CVPR.2001.990517 http://dx.doi.org/10.1109/CVPR.2001.990517 ]

Wang H T, Li S Z, Wang Y S and Zhang J J. 2004. Self quotient image for face recognition//Proceedings of 2004 International Conference on Image Processing. Singapore, Singapore: IEEE: 1397-1400[ DOI:10.1109/ICIP.2004.1419763 http://dx.doi.org/10.1109/ICIP.2004.1419763 ]

Wang L and Sun R S. 2012. Study on face feature recognition-based fatigue monitoring method for air traffic controller. China Safety Science Journal, 22(7):66-71

汪磊, 孙瑞山. 2012.基于面部特征识别的管制员疲劳监测方法研究.中国安全科学学报, 22(7):66-71[DOI:10.3969/j.issn.1003-3033.2012.07.011]

Zhang K P, Zhang Z P, Li Z F and Qiao Y. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499-1503[DOI:10.1109/LSP.2016.2603342]

Zhang X B, Cheng B and Feng R J. 2010. Real-time detection of driver drowsiness based on steering performance. Journal of Tsinghua University (Science and Technology),50(7):1072-1076, 1081

张希波, 成波, 冯睿嘉. 2010.基于方向盘操作的驾驶人疲劳状态实时检测方法.清华大学学报(自然科学版), 50(7):1072-1076, 1081[DOI:10.16511/j.cnki.qhdxxb.2010.07.025]