浏览全部资源
扫码关注微信
收稿：2021-06-15，
修回：2021-10-31，
录用：2021-11-7，
纸质出版：2022-11-16
移动端阅览
目的
2
抑郁症是一种常见的情感性精神障碍，会带来诸多情绪和身体问题。在实践中，临床医生主要通过面对面访谈并结合自身经验评估抑郁症的严重程度。这种诊断方式具有较强的主观性，整个过程比较耗时，且易造成误诊、漏诊。为了客观便捷地评估抑郁症的严重程度，本文围绕面部图像研究深度特征提取及其在抑郁症自动识别中的应用，基于人脸图像的全局和局部特征，构建一种融合通道层注意力机制的多支路卷积网络模型，进行抑郁症严重程度的自动识别。
方法
2
首先从原始视频提取图像，使用多任务级联卷积神经网络检测人脸关键点。在对齐后分别裁剪出整幅人脸图像和眼睛、嘴部区域图像，然后将它们分别送入与通道层注意力机制结合的深度卷积神经网络以提取全局特征和局部特征。在训练时，将训练图像进行标准化预处理，并通过翻转、裁剪等操作增强数据。在特征融合层将3个支路网络提取的特征拼接在一起，最后输出抑郁症严重程度的分值。
结果
2
在AVEC 2013（The Continuous Audio/Visual Emotion and Depression Recognition Challenge）抑郁症数据库上平均绝对误差为6.74、均方根误差为8.70，相较于Baseline分别降低4.14和4.91；在AVEC2014抑郁症数据库上平均绝对误差和均方根误差分别为6.56和8.56，相较于Baseline分别降低2.30和2.30。同时，相较于其他抑郁症识别方法，本文方法取得了最低的平均绝对误差和均方根误差。
结论
2
本文方法能够以端到端的形式实现抑郁症的自动识别，将特征提取和抑郁症严重程度识别在统一框架下进行和调优，学习到的多种视觉特征更加具有鉴别性，实验结果表明了该算法的有效性和可行性。
Objective
2
Depressive disorder has disabled of individuals behavior ability. According to the Diagnostic and Statistical Manual of Mental Disorders (fifth edition)
the typical symptoms of depression is likely to be low mood
loss of interest and lack of energy. The complicated somatic symptoms are manifested inattention
insomnia
slow reaction time
reduced activity and fatigue. Recent assessment of depression severity mainly depends on the interview between the qualified psychiatrist and the patient or interview with their family member or caregiver
combined with their own prior experience. However
depression is a heterogeneous disease derived from different causes and manifestations. Basically
this diagnosis is relatively subjective and unidentified
which is lack of standardization to measure it. Current researches have revealed that the symptoms of depression patients are usually presented through a variety of visual signals
including the patient's facial expressions and body postures
especially the former are significant indicators of depression. Emerging machine learning technique can be used to capture the subtle expression changes
which can be applied to the automatic assessment of depression severity further. The early symptoms of depression can be clarified in time
and even independent diagnostic evaluation can be carried out. In order to achieve clearer evaluation of depression severity
depth features extraction and its application in automatic recognition of depression has been investigated based on facial images.
Method
2
In order to obtain effective face images-derived global information and make full use of the emotion-rich local areas like eyes and mouth parts in face images
we develop a channel-wise attention mechanism- integrated multi-branches convolutional neural network to extract multiple visual features for automatic depression recognition. First
the original video is sampled every 10 frames to reduce the redundancy of video frames
and the multi-task cascade convolutional neural networks are used to detect and locate the key points of the face. According to the coordinates of the detected key points
the areas of the whole face
eyes and mouth are cropped conversely. Then
1) to obtain global features
the completed face images are input into a deep convolutional neural network in combination with channel-wise attention mechanism
and 2) to obtain local features
the latter are input into another two deep convolutional neural networks as well. During the training
the images are preprocessed and normalized and the data is enhanced by flipping and clipping. At the feature fusion layer
the features are extracted from three branches networks concatenated. Finally
the fully connected layer outputs the depression score. To demonstrate the feasibility and reliability of our proposal
a series of experiments are performed on the commonly-used The Continuous Audio/Visual Emotion and Depression Recognition Challenge 2013(AVEC 2013) and AVEC2014 datasets.
Result
2
1) The AVEC2013 depression database: the mean absolute error (MAE) is 6.74 and the root mean square error (RMSE) is 8.70
which decreased by 4.14 and 4.91 in comparison with the Baseline; 2) the AVEC2014 depression database: the MAE and RMSE are 6.56 and 8.56 of each
which decreased by 2.30 and 2.30 in comparison with the Baseline. Furthermore
our algorithms optimize the MAE and RMSE values further on both databases. Experimental results show that the channel-wise attention mechanism not only speeds up the convergence of network
but also reduces the results of MAE and RMSE. To recognize its depression severity
the integration of eyes and mouth features can significantly decrease the error compared to using global features only. This suggests that the local features in the context of eyes and mouth is an effective supplement to global features in depression recognition
especially the salient information-related eyes conveying. It conforms to clinicians' experience in judging the depression severity.
Conclusion
2
We develop a multi-features-based automatic depression recognition method through using multi-branches deep conventional neural networks
which is implemented in an end-to-end manner. The features extraction and depression recognition are carried out and optimized in terms of a coordinated framework. To model more distinctive features
the channel-wise attention mechanism is added to adaptively assign weights to channels.
Al Jazaery M and Guo G D. 2021. Video-based depression level analysis by encoding deep spatiotemporal features. IEEE Transactions on Affective Computing, 12(1): 262-268[DOI: 10.1109/TAFFC.2018.2870884]
Beck A T, Steer R A, Ball R and Ranieri W F. 1996. Comparison of beck depression inventories-IA and-Ⅱ in psychiatric outpatients. Journal of Personality Assessment, 67(3): 588-597[DOI: 10.1207/s15327752jpa6703_13]
Chmielewski M, Clark L A, Bagby R M and Watson D. 2015. Method matters: understanding diagnostic reliability in DSM-Ⅳ and DSM-5. Journal of Abnormal Psychology, 124(3): 764-769[DOI: 10.1037/abn0000069]
De Melo W C, Granger E and Hadid A. 2022. A deep multiscale spatiotemporal network for assessing depression from facial dynamics. IEEE Transactions on Affective Computing, 13(3): 1581-1592[DOI: 10.1109/TAFFC.2020.3021755]
De Melo, W C, Granger E and Lopez M B. 2020. Encoding temporal information for automatic depression recognition from facial analysis//Proceedings of ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Barcelona, Spain: IEEE: 1080-1084[ DOI: 10.1109/ICASSP40776.2020.9054375 http://dx.doi.org/10.1109/ICASSP40776.2020.9054375 ]
Ellgring H. 1989. Non-verbal Communication in Depression. Cambridge: Cambridge University Press
He L, Jiang D M and Sahli H. 2019. Automatic depression analysis using dynamic facial appearance descriptor and dirichlet process fisher encoding. IEEE Transactions on Multimedia, 21(6): 1476-1486[DOI: 10.1109/TMM.2018.2877129]
Huang Y Q, Wang Y, Wang H, Liu Z R, Yu X, Yan J, Yu Y Q, Kou C G, Xu X F, Lu J, Wang Z Z, He S L, Xu Y F, He Y L, Li T, Guo W J, Tian H J, Xu G M, Xu X D, Ma Y J, Wang L H, Wang L M, Yan Y P, Wang B, Xiao S Y, Zhou L, Li L J, Tan L W, Zhang T T, Ma C, Li Q, Ding H, Geng H C, Jia F J, Shi J F, Wang S L, Zhang N, Du X B, Du X D and Wu Y. 2019. Prevalence of mental disorders in China: a cross-sectional epidemiological study. The Lancet Psychiatry, 6(3): 211-224[DOI: 10.1016/S2215-0366(18)30511-X]
Jan A, Meng H Y, Gaus Y F A, Zhang F and Turabzadeh S. 2014. Automatic depression scale prediction using facial expression dynamics and regression//Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge. Orlando, USA: Association for Computing Machinery: 73-80[ DOI: 10.1145/2661806.2661812 http://dx.doi.org/10.1145/2661806.2661812 ]
Jan A, Meng H Y, Gaus Y F B A and Zhang F. 2018. Artificial intelligent system for automatic depression level analysis through visual and vocal expressions. IEEE Transactions on Cognitive and Developmental Systems, 10(3): 668-680[DOI: 10.1109/TCDS.2017.2721552]
Jiang X, Shao Z H, Shang Y Y and Ding H. 2019. Depression recognition based on cascaded deep neural networks. Computer Applications and Software, 36(10): 117-122, 150
江筱, 邵珠宏, 尚媛园, 丁辉. 2019. 基于级联深度神经网络的抑郁症识别. 计算机应用与软件, 36(10): 117-122, 150[DOI: 10.3969/j.issn.1000-386x.2019.10.021]
Lynch V A and Duval J B. 2010. Forensic Nursing Science. Elsevier[EB/OL]. [2021-06-15] . https://ieeexplore.ieee.org/document/8976305 https://ieeexplore.ieee.org/document/8976305
Ma X C, Huang D, Wang Y H and Wang Y D. 2016.Cost-sensitive two-stage depression prediction using dynamic visual clues//Proceedings of the 13th Asian Conference on Computer Vision. Taipei, China: Springer: 338-351[ DOI: 10.1007/978-3-319-54184-6_21 http://dx.doi.org/10.1007/978-3-319-54184-6_21 ]
Meng H Y, Huang D, Wang H, Yang H Y, Ai-Shuraifi M and Wang Y H. 2013. Depression recognition based on dynamic facial and vocal expression features using partial least square regression//Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. Barcelona, Spain: Association for Computing Machinery: 21-30[ DOI: 10.1145/2512530.2512532 http://dx.doi.org/10.1145/2512530.2512532 ]
Niu M Y, Tao J H and Liu B. 2019a. Local second-order gradient cross pattern for automatic depression detection//Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos. Cambridge, UK: IEEE: 128-132[ DOI: 10.1109/ACⅡW.2019.8925158 http://dx.doi.org/10.1109/ACⅡW.2019.8925158 ]
Niu M Y, Tao J H, Liu B and Fan C H. 2019b. Automatic depression level detection via $$\ell_p$$ -norm pooling//Proceedings of the INTERSPEECH 2019. Graz, Austria: ISCA: 4559-4563[ DOI: 10.21437/interspeech.2019-1617 http://dx.doi.org/10.21437/interspeech.2019-1617 ].
Niu M Y, Tao J H, Liu B, Huang J and Lian Z. 2020. Multimodal spatiotemporal representation for automatic depression level detection. IEEE Transactions on Affective Computing: #3031345[DOI: 10.1109/TAFFC.2020.3031345]
Pampouchidou A, Pediaditis M, Kazantzaki E, Sfakianakis S, Apostolaki I A, Argyraki K, Manousos D, Meriaudeau F, Marias K, Yang F, Tsiknakis M, Basta M, Vgontzas A N and Simos P. 2020. Automated facial video-based recognition of depression and anxiety symptom severity: cross-corpus validation. Machine Vision and Applications, 31(4): #30[DOI: 10.1007/s00138-020-01080-7]
Pampouchidou A, Simos P G, Marias K, Meriaudeau F, Yang F, Pediaditis M and Tsiknakis M. 2019. Automatic assessment of depression based on visual cues: a systematic review. IEEE Transactions on Affective Computing, 10(4): 445-470[DOI: 10.1109/TAFFC.2017.2724035]
Song S Y, Jaiswal S, Shen L L and Valstar M. 2022. Spectral representation of behaviour primitives for depression analysis. IEEE Transactions on Affective Computing, 13(2): 829-844[DOI: 10.1109/TAFFC.2020.2970712]
Sotelo J L and Nemeroff C B. 2017. Depression as a systemic disease. Personalized Medicine in Psychiatry, 1-2: 11-25[DOI: 10.1016/j.pmip.2016.11.002]
Uddin A, Joolee J B and Lee Y K. 2022. Depression level prediction using deep spatiotemporal features and multilayer Bi-LTSM. IEEE Transactions on Affective Computing, 13(2): 864-870[DOI: 10.1109/TAFFC.2020.2970418]
Valstar M, Schuller B, Smith K, Almaev T, Eyben F, Krajewski J, Cowie R and Pantic M. 2014. AVEC 2014: 3D dimensional affect and depression recognition challenge//Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge. Orlando, USA: Association for Computing Machinery: 3-10[ DOI: 10.1145/2661806.2661807 http://dx.doi.org/10.1145/2661806.2661807 ]
Valstar M, Schuller B, Smith K, Eyben F, Jiang B H, Bilakhia S, Schnieder S, Cowie R and Pantic M. 2013. AVEC 2013: the continuous audio/visual emotion and depression recognition challenge//Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. Barcelona, Spain: Association for Computing Machinery: 3-10[ DOI: 10.1145/2512530.2512533 http://dx.doi.org/10.1145/2512530.2512533 ]
Wen L Y, Li X, Guo G D and Zhu Y. 2015. Automated depression diagnosis based on facial dynamic analysis and sparse coding. IEEE Transactions on Information Forensics and Security, 10(7): 1432-1441[DOI: 10.1109/TIFS.2015.2414392]
Williamson J R, Quatieri T F, Helfer B S, Ciccarelli G and Mehta D D. 2014. Vocal and facial biomarkers of depression based on motor incoordination and timing//Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge. Orlando, USA: Association for Computing Machinery: 65-72[ DOI: 10.1145/2661806.2661809 http://dx.doi.org/10.1145/2661806.2661809 ]
Yi D, Lei Z, Liao S C and Li S Z. 2014. Learning face representation from scratch[EB/OL]. [2021-06-15] . https://arxiv.org/pdf/1411.7923.pdf https://arxiv.org/pdf/1411.7923.pdf
Zhang K P, Zhang Z P, Li Z F and Qiao Y. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10): 1499-1503[DOI: 10.1109/LSP.2016.2603342]
Zhao Z P, Li Q F, Cummins N, Liu B, Wang H S, Tao J H and Schuller B W. 2020. Hybrid network feature extraction for depression assessment from speech//Proceedings of the INTERSPEECH 2020. Shanghai, China: International Speech Communication Association: 4956-4960[ DOI: 10.21437/Interspeech.2020-2396 http://dx.doi.org/10.21437/Interspeech.2020-2396 ]
Zhou X Z, Jin K, Shang Y Y and Guo G D. 2020. Visually interpretable representation learning for depression recognition from facial images. IEEE Transactions on Affective Computing, 11(3): 542-552[DOI: 10.1109/TAFFC.2018.2828819]
Zhu Y, Shang Y Y, Shao Z H and Guo G D. 2018. Automated depression diagnosis based on deep networks to encode facial appearance and dynamics. IEEE Transactions on Affective Computing, 9(4): 578-584[DOI:10.1109/TAFFC.2017.2650899]
相关作者
相关机构
京公网安备11010802024621
