浏览全部资源
扫码关注微信
Received：21 July 2017，
Revised：2017-11-30，
Published：16 May 2018
移动端阅览
目的
2
针对当前视频情感判别方法大多仅依赖面部表情、而忽略了面部视频中潜藏的生理信号所包含的情感信息，本文提出一种基于面部表情和血容量脉冲（BVP）生理信号的双模态视频情感识别方法。
方法
2
首先对视频进行预处理获取面部视频；然后对面部视频分别提取LBP-TOP和HOG-TOP两种时空表情特征，并利用视频颜色放大技术获取BVP生理信号，进而提取生理信号情感特征；接着将两种特征分别送入BP分类器训练分类模型；最后利用模糊积分进行决策层融合，得出情感识别结果。
结果
2
在实验室自建面部视频情感库上进行实验，表情单模态和生理信号单模态的平均识别率分别为80%和63.75%，而融合后的情感识别结果为83.33%，高于融合前单一模态的情感识别精度，说明了本文融合双模态进行情感识别的有效性。
结论
2
本文提出的双模态时空特征融合的情感识别方法更能充分地利用视频中的情感信息，有效增强了视频情感的分类性能，与类似的视频情感识别算法对比实验验证了本文方法的优越性。另外，基于模糊积分的决策层融合算法有效地降低了不可靠决策信息对融合的干扰，最终获得更优的识别精度。
Objective
2
With the continuous development of artificial intelligence
researchers and scholars from other fields have become increasingly interested in providing computers with the capability to understand the emotions conveyed by(human beings and naturally interact with them. Therefore
emotion recognition has gradually become one of the key points of research to achieve harmonious human-computer interaction. The performance of video emotion recognition algorithms critically depends on the quality of the extracted emotion information. Previous research showed that facial expression is the most direct method to convey emotional information. Thus
current works usually rely on facial expressions only to complete emotion recognition. Feature extraction methods based on facial expression images are mostly based on gray images. However
during the conversion of color images into gray images
the latent physiological signals in the color information and the hidden physiological signals contained in facial videos that have discriminant information for emotion recognition are lost. In this study
a novel dual-modality video emotion recognition method for fusion decision
which combines facial expressions and blood volume pulse (BVP) physiological signals that can be extracted from facial videos
is introduced to overcome this problem.
Method
2
First
the video is preprocessed (including face detection and normalization) to acquire a sequence of video frames that contain only the face image. The LBP-TOP feature is an effective local texture descriptor
whereas the HOG-TOP feature is a gradient-based local shape descriptor that can compensate for the lack of LBP-TOP feature extraction in image edge and direction information. Thus
in this study
we extract the LBP-TOP and HOG-TOP features from the video frames and fuse the two facial expression features. We use video color amplification technology to process the original video and extract the BVP physiological signal from the processed video. Then
the emotional feature of physiological signals can be extracted from the BVP physiological signal. Afterward
the two features are inputted into the BP classifier to train the classification models. Finally
the fuzzy integral is used to fuse the posterior probability information obtained by the two classifiers to obtain the final emotion recognition result.
Result
2
Considering that the current commonly used video emotion databases cannot satisfy the requirements for extracting the BVP signal
we conduct experimental verification by using the self-built facial expression video database. Each group of experiments was cross-validated
and the final results were averaged to increase the credibility of the experiment. The average recognition rates of single modality
i.e.
facial expression or physiological signal
are 80% and 63.75%
respectively
whereas the emotion recognition result of the fusion of the two modalities is up to 83.33%
which is higher than that of each single modality before fusion. This finding indicates that the fusion decision algorithm with facial expression and BVP physiological signal is effective for emotion recognition. The experimental results of other fusion methods
namely
the D-S evidence theory and the maximum value rule
are 71% and 80%
respectively
which are lower than that of the fuzzy integral method. In addition
the recognition rate of our method is 2% and 2.5% higher than the results of the two existing video emotion recognition methods.
Conclusion
2
The dual-modality space-time feature fusion method proposed in this study characterizes the emotion information contained in the facial videos from two aspects
i.e.
the facial expression and the physiological signals
to make full use of the emotional information of the video. The experimental results show that this algorithm can make full use of the emotion information of the video and effectively improve the classification performance of video emotion recognition. The effectiveness of our proposed method in comparison to that of similar video emotion recognition algorithms is verified. In addition
the fuzzy integral is used to fuse two different modalities at the decision level. The reliability of different classifiers in the fusion process is considered and compared with that of D-S evidence theory and the maximum value rule. The influence of unreliable decision-making information on the fusion decision is effectively reduced. Finally
a high recognition accuracy is obtained by the proposed fusion method. The contrast experiment with other fusion methods also proves the superiority of the proposed fusion method.
Cao J Q, Tong C. Facial expression recognition based on LBP-EHMM[C]//Proceedings of Congress on Image and Signal Processing. Sanya, Hainan, China: IEEE, 2008: 371-375. [ DOI:10.1109/cisp.2008.119 http://dx.doi.org/10.1109/cisp.2008.119 ]
Dahmane M, Meunier J. Emotion recognition using dynamic grid-based HoG features[C]//Proceedings of 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops. Santa Barbara, CA, USA: IEEE, 2011: 884-888. [ DOI:10.1109/fg.2011.5771368 http://dx.doi.org/10.1109/fg.2011.5771368 ]
Gu W F, Xiang C, Venkatesh Y V, et al. Facial expression recognition using radial encoding of local Gabor features and classifier synthesis[J]. Pattern Recognition, 2012, 45(1):80-91.[DOI:10.1016/j.patcog.2011.05.006]
Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):915-928.[DOI:10.1109/tpami.2007.1110]
Zhao L, Wang Z C, Zhang G X. Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and Gabor multiorientation fusion histogram[J]. Mathematical Problems in Engineering, 2017, 2017:#7206041.[DOI:10.1155/2017/7206041]
Chen J K, Chen Z H, Chi Z R, et al. Facial expression recognition in video with multiple feature fusion[J]. IEEE Transactions on Affective Computing, 2016.[DOI:10.1109/taffc.2016.2593719](in press)
Fan X J, Tjahjadi T. A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences[J]. Pattern Recognition, 2015, 48(11):3407-3416.[DOI:10.1016/j.patcog.2015.04.025]
Chang C Y, Tsai J S, Wang C J, et al. Emotion recognition with consideration of facial expression and physiological signals[C]//2009 IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology. Nashville, TN, USA: IEEE, 2009: 278-283. [ DOI:10.1109/cibcb.2009.4925739 http://dx.doi.org/10.1109/cibcb.2009.4925739 ]
Kortelainen J, Tiinanen S, Huang X H, et al. Multimodal emotion recognition by combining physiological signals and facial expressions: a preliminary study[C]//Proceedings of 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. San Diego, CA, USA: IEEE, 2012: 5238-5241. [ DOI:10.1109/embc.2012.6347175 http://dx.doi.org/10.1109/embc.2012.6347175 ]
Maaoui C, Pruski A. A comparative study of SVM kernel applied to emotion recognition from physiological signals[C]//Proceedings of the 5th International Multi-Conference on Systems, Signals and Devices. Amman, Jordan: IEEE, 2008: 1-6. [ DOI:10.1109/ssd.2008.4632891 http://dx.doi.org/10.1109/ssd.2008.4632891 ]
Kim J, André E. Emotion recognition based on physiological changes in music listening[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(12):2067-2083.[DOI:10.1109/tpami.2008.26]
Wang B. A study of bimodal emotion recognition based on facial expression and biological signals[D]. Nanjing: Southeast University, 2014. http://www.wanfangdata.com.cn/details/detail.do?_type=degree&id=Y2706490 .
王蓓. 基于面部表情和生理信号的双模态情感识别研究[D]. 南京: 东南大学, 2014.
Osman A, Turcot J, El Kaliouby R. Supervised learning approach to remote heart rate estimation from facial videos[C]//Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Ljubljana, Slovenia: IEEE, 2015: 1-6. [ DOI:10.1109/fg.2015.7163150 http://dx.doi.org/10.1109/fg.2015.7163150 ]
Poh M Z, Mcduff D J, Picard R W. Advancements in noncontact, multiparameter physiological measurements using a webcam[J]. IEEE Transactions on Biomedical Engineering, 2011, 58(1):7-11.[DOI:10.1109/tbme.2010.2086456]
Monkaresi H, Calvo R A, Yan H. A machine learning approach to improve contactless heart rate monitoring using a webcam[J]. IEEE Journal of Biomedical and Health Informatics, 2014, 18(4):1153-1160.[DOI:10.1109/jbhi.2013.2291900]
Wu H Y, Rubinstein M, Shih E, et al. Eulerian video magnification for revealing subtle changes in the world[J]. ACM Transactions on Graphics, 2012, 31(4):#65.[DOI:10.1145/2185520.2185561]
Fu Y W, Li X, Zhuang Z W. A decision level fusion algorithm for target recognition with fuzzy densities determined adaptively[J]. Acta Electronica Sinica, 2004, 32(9):1433-1435.
付耀文, 黎湘, 庄钊文.一种自适应模糊密度赋值的决策层融合目标识别算法[J].电子学报, 2004, 32(9):1433-1435. [DOI:10.3321/j.issn:0372-2112.2004.09.006]
相关作者
相关机构
京公网安备11010802024621
