深度空时婴幼儿表情识别模型下的ASD自动筛查

唐传高; 郑文明; 宗源; 仇娜娜; 闫思蒙; 翟梦瑶; 柯晓燕

doi:10.11834/jig.200360

精神状况评估 | 浏览量 : 0 下载量: 52 CSCD: 1

PDF
导出
分享
收藏
专辑

深度空时婴幼儿表情识别模型下的ASD自动筛查
Automated early identification of ASD under the deep spatiotemporal model-based facial expression analysis for infants and toddlers
2020年25卷第11期页码：2404-2414
收稿：2020-07-09，

修回：2020-9-25，

录用：2020-10-2，

纸质出版：2020-11-16
DOI： 10.11834/jig.200360
稿件说明：

移动端阅览

唐传高, 郑文明, 宗源, 仇娜娜, 闫思蒙, 翟梦瑶, 柯晓燕. 深度空时婴幼儿表情识别模型下的ASD自动筛查[J]. 中国图象图形学报, 2020,25(11):2404-2414. DOI： 10.11834/jig.200360.

Chuangao Tang, Wenming Zheng, Yuan Zong, Nana Qiu, Simeng Yan, Mengyao Zhai, Xiaoyan Ke. Automated early identification of ASD under the deep spatiotemporal model-based facial expression analysis for infants and toddlers[J]. Journal of Image and Graphics, 2020, 25(11): 2404-2414. DOI： 10.11834/jig.200360.

摘要

目的

高危孤独症谱系障碍（high-risk autism spectrum disorder，HR-ASD）筛查依赖于医师的临床评估和问卷量表，传统筛查方式效率低，亟需一种高效的自动筛查工具。为了满足自动筛查的需求，本文提出一种基于婴幼儿表情分析的孤独症谱系障碍自动筛查方法。

方法

首先入组30例8~18个月的婴幼儿，包括10例ASD疑似患儿（HR-ASD）和20例正常发育婴幼儿；引入静止脸范式，并利用该范式诱发婴幼儿在社交压力条件下的情绪调节行为；提出一种面向婴幼儿视频表情识别的深度空时特征学习网络，首先在大规模公开数据集AffectNet预训练空域特征学习模型，然后在自建婴幼儿面部表情视频数据集RCLS&NBH+（Research Center of Learning Science&Nanjing Brain Hospital dataset+）上训练时空特征学习模型，从而建立一个较精准的婴幼儿表情识别模型；基于该模型深度特征序列的一阶统计量，构建婴幼儿社交压力环境下的表情行为症状与精神健康状态之间的关联，采用机器学习方法实现自动筛查。

结果

1）基于婴幼儿表情人工标注的结果，发现：在1 min静止期，高危组的婴幼儿中性表情持续时长相对正常对照组偏高（

0.01），而其他表情未发现有统计学意义的差异；2）提出的深度空时特征学习网络在本研究的30例婴幼儿面部表情视频数据集上的总体平均识别率达到了87.1%，3类表情预测结果与人工标注结果具有较高的一致性，其中Kappa一致性系数达到0.63，Pearson相关系数达到0.67；3）基于面部表情深度特征序列一阶统计量的精神健康状态预测性能达到灵敏度70%，特异性90%，分类正确率83.3%（置换检验

0.05）。

结论

本文提出的基于婴幼儿面部表情深度特征序列一阶统计量的精神健康状态自动预测模型是有效的，有助于实现高危孤独症谱系障碍自动筛查。

Abstract

Objective

The traditional screening of high-risk autism spectrum disorder (HR-ASD) mainly relies on the evaluation of pediatric clinicians. Owing to the low efficiency of this method

highly efficient automated screening tools have become a major research topic. Although most facial expression markers-related research have achieved some progress

the findings were derived from comparatively older children

and the effectiveness of the paradigm used in these research has not been tested in babies aged 8~18 months

whose intelligence quotient (IQ)

language

and social ability are still developing. In these studies

a lack of a final diagnostic model led to low feasibility values in the large-scale screening of ASD. In this study

a novel automated screening method that provide diagnostic results based on the analysis of babies' facial expression symptoms under social stress environments was proposed.

Method

Differences among the babies' facial expressions in the HR-ASD and typically developing (TD) comparison groups were determined. A total of 30 infants and toddlers were enrolled in our study

of which 10 were at risk of HR-ASD and 20 were TD babies. All the babies were 8~18 months at the time of enrollment

all the participants received a re-diagnosis during 25 months of life in order that each case in the two groups is true ASD or TD. The still-face paradigm

including an amusing mother-baby interaction episode (baseline

2 min) and a still-face episode (1 min)

was employed to induce babies' emotion regulation behaviors for social stress environment in subsequent episode. We hypothesized that facial features derived from an accurate facial expression recognition system can be used in distinguishing the two groups. This hypothesis was then verified. For the establishment of an accurate facial expression recognition system

a deep spatiotemporal feature learning network was proposed. The spatial feature learning module was pretrained on an open-access dataset named AffectNet and was further trained on a video-based baby facial expression dataset

Research Center of Learning Science

Nanjing Brain Hospital dataset+(RCLS

NBH+ 53 babies subjects

101 videos

and 95 207 babies' facial images)

and a bidirectional long-short term memory network (Bi-LSTM) was used. The trained deep spatiotemporal neural network was verified using the collected babies' facial expression dataset including 30 babies

that is

the infant emotion dataset (IED). Three types of learned features derived from deep neural networks

including feature_a (the output of the last fully connected layer with 1 024 units in a convolutional neural network (CNN) that was only trained on the AffectNet dataset)

feature_b (feature_a's counterpart in the CNN part of the CNN+LSTM(long short term memory) model)

and feature_c (the output containing 1 024 units derived from the Bi-LSTM module)

were compared. Pearson's correlation was computed between these frame-level learned features and their corresponding frame-level facial expression labels. Feature subsets were selected using different correlation thresholds

including without threshold (1 024-d features)

0.2

0.4

and 0.6

1. Then

the use of first-order statistical measurement

that is

frame-level mean values of selected features within a video was proposed in exploring the association between babies' mental health status and their facial expression symptoms under social stress environment. Such features were fed to linear classifiers for the automated screening of HR-ASD.

Result

1) Basing on the human coding for babies' facial expressions under the still-face paradigm

we find that babies at a high risk of ASD showed more neutral facial expressions (55.03±7.34 s) than those in the TD comparison group (46.26±11.02 s) during the one-minute still-face episode (

0.01). The other two types of facial expressions (positive and negative facial expressions) did not show statistically significant differences; 2) The proposed deep spatiotemporal neural network achieved an overall average recognition accuracy of 87.1% on a self-collected infants and toddlers' facial expression dataset

IED

which included 30 babies in this study. The recall rates for positive

neutral

and negative facial expressions were 68.82%

93.79%

and 59.57%

respectively

whereas the recall rates for positive

neutral and negative facial expressions corresponding to the spatial model just trained on AffectNet were 36.32%

77.06%

and 58.42%

respectively. A high consistency between the automated emotion prediction results from the CNN+LSTM model and human coding results was found

that is

a Kappa coefficient of 0.63 and Pearson coefficient of 0.67 were attained. 3) Through a leave-one subject out cross-validation

a sensitivity of 70%

specificity of 90%

and overall diagnostic accuracy of 83.3%

where the

-value of permutation test was lower than 0.05

were achieved using the proposed automated screening model based on the linear discriminant classifier and proposed features (feature_c derived from the CNN+LSTM model) under the correlation threshold of 0.6

which also verified the proposed hypothesis

and a more accurate facial expression recognition model showed better diagnostic performance between HR-ASD and TD according to the comparison between feature_a and feature_c.

Conclusion

The automated screening method based on the proposed features from babies' facial expression is effective

showing potential for large-scale applications.

关键词

Keywords

references

Baltrusaitis T, Zadeh A, Lim Y C and Morency L P. 2018. Openface 2.0: facial behavior analysis toolkit//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Xi'an: IEEE: 59-66[ DOI: 10.1109/FG.2018.00019 http://dx.doi.org/10.1109/FG.2018.00019 ]

Bosl W J, Tager-Flusberg H and Nelson C A. 2018. EEG analytics for early detection of autism spectrum disorder:a data-driven approach. Scientific Reports, 8(1):#6828[DOI:10.1038/s41598-018-24318-x]

Estes A, Zwaigenbaum L, Gu H B, John T S, Paterson S, Elison J T, Hazlett H, Botteron K, Dager S R, Schultz R T, Kostopoulos P, Evans A, Dawson G, Eliason J, Alvarez S, Piven J and IBIS network. 2015. Behavioral, cognitive, and adaptive development in infants with autism spectrum disorder in the first 2 years of life. Journal of Neurodevelopmental Disorders, 7(1):#24[DOI:10.1186/s11689-015-9117-6]

Guha T, Yang Z J, Grossman R B and Narayanan S S. 2018. A computational study of expressive facial dynamics in children with autism. IEEE Transactions on Affective Computing, 9(1):14-20[DOI:10.1109/TAFFC.2016.2578316]

Hammal Z, Chu W S, Cohn J F, Heike C and Speltz M L. 2017. Automatic action unit detection in infants using convolutional neural network//Proceedings of the 7th International Conference on Affective Computing and Intelligent Interaction (ACII). San Antonio: IEEE: 216-221[ DOI: 10.1109/ACII.2017.8273603 http://dx.doi.org/10.1109/ACII.2017.8273603 ]

Hashemi J, Dawson G, Carpenter K L H, Campbell K, Qiu Q, Espinosa S, Marsan S, Baker J P, Egger H L and Sapiro G. 2018. Computer vision analysis for quantification of autism risk behaviors. IEEE Transactions on Affective Computing: #8453869[ DOI: 10.1109/TAFFC.2018.2868196 http://dx.doi.org/10.1109/TAFFC.2018.2868196 ]

Ma W N, Zhu B B and Xie Y. 2015. Facial emotion recognition in children with autism spectrum disorder:evidence from eye movements. Chinese Journal of Applied Psychology, 21(1):76-86

马伟娜, 朱蓓蓓, 谢宇. 2015.孤独症儿童面部表情识别能力的眼动研究.应用心理学, 21(1):76-86[DOI:10.3969/j.issn.1006-6020.2015.01.009]

Manfredonia J, Bangerter A, Manyakov N V, Ness S, Lewin D, Skalkin A, Boice M, Goodwin M S, Dawson G, Hendren R, Leventhal B, Shic F and Pandina G. 2019. Automatic recognition of posed facial expression of emotion in individuals with autism spectrum disorder. Journal of Autism and Developmental Disorders, 49(1):279-293[DOI:10.1007/s10803-018-3757-9]

Mollahosseini A, Hasani B and Mahoor M H. 2019. Affectnet:a database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18-31[DOI:10.1109/TAFFC.2017.2740923]

Qiu N N, Tang C G, Zhai M Y, Huang W Q, Weng J, Li C Y, Xiao X, Fu J L, Zhang L L, Xiao T, Fang H and Ke X Y. 2020. Application of the still-face paradigm in early screening for high-risk autism spectrum disorder in infants and toddlers. Frontiers in Pediatrics, 8:#290[DOI:10.3389/fped.2020.00290]

Qiu N N, Zhai M Y, Tang C G, Weng J, Feng M, Xiao X, Xiao T, Li C Y, Wang Y, Jin PY, Wang Y, Da Y, Zheng W M and Ke X Y. 2019. The makers of early social behavior in toddlers with high-risk of autism spectrum disorder:an observational pilot study. Chinese Journal of Psychiatry, 52(1):50-56

仇娜娜, 翟梦瑶, 唐传高, 翁娇, 封敏, 肖湘, 肖婷, 李春燕, 王瑶, 金佩莹, 王月, 笪云, 郑文明, 柯晓燕. 2019.高危孤独症谱系障碍幼儿早期社会性行为学指标的研究.中华精神科杂志, 52(1):50-56 [DOI:10.3760/cma.j.issn.1006-7884.2019.01.003]

Samad M D, Diawara N, Bobzien J L, Harrington J W, Witherow M A and Iftekharuddin K M. 2018. A feasibility study of autism behavioral markers in spontaneous facial, visual, and hand movement response data. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(2):353-361[DOI:10.1109/TNSRE.2017.2768482]

Sarabadani S, Schudlo L C, Samadani A A and Kushki A. 2018. Physiological detectionof affective states in children with autism spectrum disorder. IEEE Transactions on Affective Computing: #8326711[ DOI: 10.1109/TAFFC.2018.2820049 http://dx.doi.org/10.1109/TAFFC.2018.2820049 ]

Sariyanidi E, Gunes H and Cavallaro A. 2015. Automatic analysis of facial affect:a survey of registration, representation, and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(6):1113-1133[DOI:10.1109/TPAMI.2014.2366127]

Wang F F, Li X, Liu J and Ji Z Z. 2019. Emotional and behavioral problems in toddlers with autism. Chinese General Practice, 22(18):2189-2193

王菲菲, 李雪, 刘靖, 吉兆正. 2019.孤独症幼儿的情绪行为问题研究.中国全科医学, 22(18):2189-2193

Wang L, Li G, Shi F, Cao X H, Lian C F, Nie D, Liu M X, Zhang H, Li G N, Wu Z W, Lin W L and Shen D G. 2018. Volume-based analysis of 6-month-old infant brain MRI for autism biomarker identification and early diagnosis//Proceedings of the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention. Granada, Spain: Springer: 411-419[ DOI: 10.1007/978-3-030-00931-1_47 http://dx.doi.org/10.1007/978-3-030-00931-1_47 ]

Yirmiya N, Gamliel I, Pilowsky T, Feldman R, Baron-Cohen S and Sigman M. 2006. The development of siblings of children with autism at 4 and 14 months:social engagement, communication, and cognition. Journal of Child Psychology and Psychiatry, 47(5):511-523[DOI:10.1111/j.1469-7610.2005.01528.x]

Zhai M Y, Qiu N N, Tang C G, Weng J, Zhang L L, Feng M, Xiao X, Xiao T, Zheng W M and Ke X Y. 2020. Characteristics of vocalization in children with autism spectrum disorder during the still-face paradigm. Chinese Journal of Contemporary Pediatrics, 22(4):361-367

翟梦瑶, 仇娜娜, 唐传高, 翁娇, 章丽丽, 封敏, 肖湘, 肖婷, 郑文明, 柯晓燕. 2020.孤独症谱系障碍患儿静止脸试验发声行为特征分析.中国当代儿科杂志, 22(4):361-367[DOI:10.7499/j.issn.1008-8830.2001006]

Zhang K P, Zhang Z P, Li Z F and Qiao Y. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499-1503[DOI:10.1109/LSP.2016.2603342]

文章被引用时，请邮件提醒。

提交

暂无数据