浏览全部资源
扫码关注微信
纸质出版日期： 2021-12-16 ，
录用日期： 2021-02-10
移动端阅览
刘婷婷, 刘箴, 柴艳杰, 王瑾, 王媛怡. 人机交互中的智能体情感计算研究[J]. 中国图象图形学报, 2021,26(12):2767-2777.
Tingting Liu, Zhen Liu, Yanjie Chai, Jin Wang, Yuanyi Wang. Agent affective computing in human-computer interaction[J]. Journal of Image and Graphics, 2021,26(12):2767-2777.
机器的情感是通过融入具有情感能力的智能体实现的，虽然目前在人机交互领域已经有大量研究成果，但有关智能体情感计算方面的研究尚处起步阶段，深入开展这项研究对推动人机交互领域的发展具有重要的科学和应用价值。本文通过检索Scopus数据库选择有代表性的文献，重点关注情感在智能体和用户之间的双向流动，分别从智能体对用户的情绪感知和对用户情绪调节的角度开展分析总结。首先梳理了用户情绪的识别方法，即通过用户的表情、语音、姿态、生理信号和文本信息等多通道信息分析用户的情绪状态，归纳了情绪识别中的一些机器学习方法。其次从用户体验角度分析具有情绪表现力的智能体对用户的影响，总结了智能体的情绪生成和表现技术，指出智能体除了通过表情之外，还可以通过注视、姿态、头部运动和手势等非言语动作来表现情绪。并且梳理了典型的智能体情绪架构，举例说明了强化学习在智能体情绪设计中的作用。同时为了验证模型的准确性，比较了已有的情感评估手段和评价指标。最后指出智能体情感计算急需解决的问题。通过对现有研究的总结，智能体情感计算研究是一个很有前景的研究方向，希望本文能够为深入开展相关研究提供借鉴。
Human computer interaction technology has been promoting to realize intelligent human-computer interaction. The user's emotional experience in the human-computer interaction system has been facilitated based on the realization of emotional interaction. Emotional interaction has been intended to use widely via Gartner's analysis next decade. The agent can be real or virtual to detect the user's emotion and adjust the user's emotion. It can greatly enhance the user's experience in human-computer interaction on the aspects of psychological rehabilitation
E-education
digital entertainment
smart home
virtual tourism
E-commerce and etc. The research of agent's affective computing has involved in computer graphics
virtual reality
human-computer-based interaction
machine learning
psychology
social science. Based on Scopus database
2 080 journal papers have been optioned via using virtual human (agent
multimodal) plus emotional interaction as the key words each. The perspective of agent's perceptions and influence of users' emotions have been analyzed and summarized. The importance of multi-channel in emotion perception and the typical machine learning algorithms in emotion recognition have been summarized from the perspective of agent's perception of users' emotions. The external and internal factors affecting users' emotions have been analyzed from the perspective of agent's influence on users' emotions. The emotional architecture
emotional generation and expression algorithms have been implemented. Customized evaluation methods have been applied to improve the accuracy of the affective computing algorithm. The importance of emotional agent in human-computer interaction has been analyzed. Four key steps of agent affective computing have been summarized based on current studies: 1) An agent expressed its emotion to the user. 2) The user gave their feedback to the agent (they may or may not express their satisfaction or dissatisfaction via some channels like facial expressions). 3) The real-time agent perceived the user's emotional state and intention and adjustable emotional performance to respond to user's feedback. 4) A standard (e.g.
the completion of emotion regulation task
the end of plot) has been reached
the agent stopped interacting with the user
otherwise
returns to step 1). The current studies have shown that user's expressed emotions via facial expressions
voices
postures
physiological signals and texts on the aspect of user emotion recognition. The multi-channel method has been more reliable than the single channel method. Machine learning can be used to extract emotional features. Typical machine learning algorithms and their applicable scenarios have been sorted out based on CNN (convolutional neural network) nowadays. Some solutions have been facilitated to resolve insufficient data and over fitting issues. Spatial distance
the number of agents
the appearance of the agent
brightness and shadow have been set as external factors. Agent's autonomous emotion expression has been targeted as the internal factor. An agent should have an emotional architecture and use facial expression
eye gaze
posture
head movement gesture and other channels to express its emotion. The accuracy of the emotional classification model and users' feelings has been assessed based on an affective computing model. The statistical sampling analysis has been listed in the table. The existing emotional agents such as low intelligence
lack of common-sense database
lack of interactivity have been as the constrained factors. The research of agent affective computing in the field of human-computer interaction has been developed further. An affective computing for human-computer-based interaction and an agent could be a channel of emotional interaction. Knowledge-based database and appropriated machine learning algorithms have been adopted to build an agent with the ability of emotion perception and emotion regulation. Qualified physiological detection equipment and sustainable enrichment of emotional information assessment methods have developed the affective computing further.
人机交互智能体情感计算情绪感知表现
human-computer interactionagentaffective computingemotionperceptionperformance
Ajili I, Mallem M and Didier J Y. 2019. Human motions and emotions recognition inspired by LMA qualities. The Visual Computer, 35(10): 1411-1426 [DOI: 10.1007/s00371-018-01619-w]
Amini R, Lisetti C and Ruiz G. 2015. HapFACS 3.0: FACS-based facial expression generator for 3D speaking virtual characters. IEEE Transactions on Affective Computing, 6(4): 348-360 [DOI: 10.1109/TAFFC.2015.2432794]
Aristidou A, Charalambous P and Chrysanthou Y. 2015. Emotion analysis and classification: understanding the performers' emotions using the LMA entities. Computer Graphics Forum, 34(6): 262-276 [DOI: 10.1111/cgf.12598]
Becker-Asano C and Wachsmuth I. 2010. Affective computing with primary and secondary emotions in a virtual human. Autonomous Agents and Multi-Agent Systems, 20(1): #32 [DOI: 10.1007/s10458-009-9094-9]
Bönsch A, Radke S, Overath H, AschéL M, Wendt J, Vierjahn T, Habel U and Kuhlen T W. 2018. Social VR: how personal space is affected by virtual agents' emotions//Proceedings of 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Tuebingen/Reutlingen, Germany: IEEE: 199-206 [DOI: 10.1109/VR.2018.8446480http://dx.doi.org/10.1109/VR.2018.8446480]
Castillo S, Wallraven C and Cunningham D W. 2014. The semantic space for facial communication. Computer Animation and Virtual Worlds, 25(3/4): 223-231 [DOI: 10.1002/cav.1593]
Chen S Z, Wang S and Jin Q. 2018. Multimodal emotion recognition in multi-cultural conditions. Journal of Software, 29(4): 1060-1070
陈师哲, 王帅, 金琴. 2018. 多文化场景下的多模态情感识别. 软件学报, 29(4): 1060-1070) [DOI: 10.13328/j.cnki.jos.005412]
D'Mello S K and Graesser A. 2010. Multimodal semi-automatedaffect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20(2): 147-187 [DOI: 10.1007/s11257-010-9074-4]
Ekman P. 1993. Facial expression and emotion. American Psychologist, 48(4): 384-392 [DOI: 10.1037/0003-066x.48.4.384]
Fan J J, Tian F, Du Y, Liu Z J and Dai G Z. 2018. Thoughts on human-computer interaction in the age of artificial intelligence. Scientia Sinica Informationis, 48(4): 361-375
范俊君, 田丰, 杜一, 刘正捷, 戴国忠. 2018. 智能时代人机交互的一些思考. 中国科学: 信息科学, 48(4): 361-375) [DOI: 10.1360/N112017-00221]
Fan X M, Fan J J, Tian F and Dai G Z. 2019. Human-computer interaction and artificial intelligence: from competition to integration. Scientia Sinica Informationis, 49(3): 361-368
范向民, 范俊君, 田丰, 戴国忠. 2019. 人机交互与人工智能: 从交替浮沉到协同共进. 中国科学: 信息科学, 49(3): 361-368) [DOI: 10.1360/N112018-00181]
Filko D and Martinović G. 2013. Emotion recognition system by a neural network based facial expression analysis. Automatika, 54(2): 263-272 [DOI: 10.7305/automatika.54-2.73]
Gratch J and Marsella S. 2004. A domain-independent framework for modeling emotion. Cognitive Systems Research, 5(4): 269-306 [DOI: 10.1016/j.cogsys.2004.02.002]
Han J X, Xu R Y and Chen L Y. 2020. Convolutional neural network fusing ranking and regression for expression intensity estimation. Journal of Computer-Aided Design and Computer Graphics, 32(8): 1228-1235
韩加旭, 徐如意, 陈靓影. 2020. 融合排序与回归的卷积神经网络用于表情强度估计. 计算机辅助设计与图形学学报, 32(8): 1228-1235) [DOI: 10.3724/SP.J.1089.2020.17753]
Jiang Y Y, Li W, Hossain M S, Chen M, Alelaiwi A and Al-Hammadi M. 2020. A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Information Fusion, 53: 209-221 [DOI: 10.1016/j.inffus.2019.06.019]
Kanjo E, Younis E M G and Ang C S. 2019. Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection. Information Fusion, 49: 46-56 [DOI: 10.1016/j.inffus.2018.09.001]
Kessous L, Castellano G and Caridakis G. 2010. Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. Journal on Multimodal User Interfaces, 3: 33-48[DOI10.1007/s12193-009-0025-5]
Ko B C. 2018. A brief review of facial emotion recognition based on visual information. Sensors, 18(2): #401 [DOI: 10.3390/s18020401]
Lee J and Marsella S C. 2010. Predicting speaker head nods and the effects of affective information. IEEE Transactions on Multimedia, 12(6): 552-562 [DOI: 10.1109/TMM.2010.2051874]
Li S and Deng W H. 2019a. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Transactions on Image Processing, 28(1): 356-370 [DOI: 10.1109/TIP.2018.2868382]
Li S and Deng W H. 2019b. Blended emotion in-the-wild: multi-label facial expression recognition using crowdsourced annotations and deep locality feature learning. International Journal of Computer Vision, 127(6): 884-906 [DOI: 10.1007/s11263-018-1131-1]
Li S and Deng W H. 2020. Deep facial expression recognition: a survey. IEEE Transactions on Affective Computing [DOI: 10.1109/TAFFC.2020.2981446http://dx.doi.org/10.1109/TAFFC.2020.2981446]
Ma Y X, Hao Y X, Chen M, Chen J C, Lu P and Košir A. 2019. Audio-visual emotion fusion (AVEF): a deep efficient weighted approach. Information Fusion, 46: 184-192 [DOI: 10.1016/j.inffus.2018.06.003]
Mao X and Xue Y L. 2011. Human-Computer Emotional Interaction. Beijing: Science Press
毛峡, 薛雨丽. 2011. 人机情感交互. 北京: 科学出版社
Moussa M B and Magnenat-Thalmann N. 2013. Toward socially responsible agents: integrating attachment and learning in emotional decision-making. Computer Animation and Virtual Worlds, 24(3-4): 327-334 [DOI: 10.1002/cav.1515]
Pferifer R and Bongard J. 2009. Body Intelligence. Yu W W, Chen W D, Yang J G, Xu M and Jin D, trans. Beijing: Science Press.
Pferifer R, Bongard J. 2009. 身体的智能: 智能科学新视角. 俞文伟, 陈卫东, 杨建国, 许敏, 金丹, 译. 北京: 科学出版社
Quan C Q, Zhang B, Sun X and Ren F J. 2017.A combined cepstral distance method for emotional speech recognition. International Journal of Advanced Robotic Systems, 14(4): 1-9 [DOI: 10.1177/1729881417719836]
Queiroz R B, Barros L M and Musse S R. 2008. Providing expressive gaze to virtual animated characters in interactive applications. Computers in Entertainment, 6(3): 1-23 [DOI: 10.1145/1394021.1394034]
Rao Y, Wu L W, Wang Y M and Feng C. 2018. Research progress on emotional computation technology based on semantic analysis. Journal of Software, 29(8): 2397-2426
饶元, 吴连伟, 王一鸣, 冯聪. 2018. 基于语义分析的情感计算技术研究进展. 软件学报, 29(8): 2397-2426) [DOI: 10.13328/j.cnki.jos.005564]
Ren F J, Yu M L, Hu M and Li Y Q. 2018. Dual-modality video emotion recognition based on facial expression and BVP physiological signal. Journal of Image and Graphics, 23(5): 688-697
任福继, 于曼丽, 胡敏, 李艳秋. 2018. 融合表情和BVP生理信号的双模态视频情感识别. 中国图象图形学报, 23(5): 688-697) [DOI: 10.11834/jig.170401]
Rosales J H, Rodríguez L F and Ramos F. 2019. A general theoretical framework for the design of artificial emotion systems in autonomous agents. Cognitive Systems Research, 58: 324-341 [DOI: 10.1016/j.cogsys.2019.08.003]
Roxas J C, Richards D, Bilgin A and Hanna N. 2018. Exploring the influence of a human-like dancing virtual character on the evocation of human emotion. Behaviour and Information Technology, 37(1): 1-15 [DOI: 10.1080/0144929X.2017.1386714]
Samara A, Galway L, Bond R and Wang H. 2019. Affective state detection via facial expression analysis within a human-computer interaction context. Journal of Ambient Intelligence and Humanized Computing, 10(6): 2175-2184 [DOI: 10.1007/s12652-017-0636-8]
Sharp H, Preece J and Rogers Y. 2003. Interaction Design. Liu X H, trans. Beijing: Publishing House of Electronics Industry
Sharp H, Preece J and Rogers Y. 2003. 交互设计: 超越人机交互. 刘晓晖, 译. 北京: 电子工业出版社
Shiratori T, Nakazawa A and Ikeuchi K. 2006. Dancing-to-music character animation. Computer Graphics Forum, 25(3): 449-458 [DOI: 10.1111/j.1467-8659.2006.00964.x]
Shu L, Xie J Y, Yang M Y, Li Z Y, Li Z Q, Liao D, Xu X M and Yang X Y. 2018. A review of emotion recognition using physiological signals. Sensors, 18(7): # 2074 [DOI: 10.3390/s18072074]
Tao J H, Yang M H, Wang Z L, Ban X J, Xie L, Wang Y H, Zeng Q, Wang F, Wang H Q, Liu B, Han Z S, Pan H and Chen W Z. 2019. Non contact multi-channel natural interactive surgical environment under sterile conditions. Journal of Software, 30(10): 2986-3004
陶建华, 杨明浩, 王志良, 班晓娟, 解仑, 汪云海, 曾琼, 王飞, 王红迁, 刘斌, 韩志帅, 潘航, 陈文拯. 2019. 无菌条件非接触式多通道自然交互手术环境. 软件学报, 30(10): 2986-3004) [DOI: 10.13328/j.cnki.jos.005785]
Tewell J, Bird J and Buchanan G R. 2017. The heat is on: a temperature display for conveying affective feedback//Proceedings of 2017 CHI Conference on Human Factors in Computing Systems. Colorado, USA: ACM: 1756-1767 [DOI: 10.1145/3025453.3025844http://dx.doi.org/10.1145/3025453.3025844]
Tinwell A, Grimshaw M, Nabi D A and Williams A. 2011. Facial expression of emotion and perception of the Uncanny Valley in virtual characters. Computers in Human Behavior, 27(2): 741-749 [DOI: 10.1016/j.chb.2010.10.018]
Volonte M, Anaraky R G, Knijnenburg B, Duchowski A T and Babu S V. 2019. Empirical evaluation of the interplay of emotion and visual attention in human-virtual human interaction//ACM Symposium on Applied Perception 2019. Barcelona, Spain: ACM: 1-9 [DOI: 10.1145/3343036.3343118http://dx.doi.org/10.1145/3343036.3343118]
Volonte M, Babu S V, Chaturvedi H, Newsome N, Ebrahimi E, Roy T, Daily S B and Fasolino T. 2016. Effects of virtual human appearance fidelity on emotion contagion in affective inter-personal simulations. IEEE Transactions on Visualization and Computer Graphics, 22(4): 1326-1335 [DOI: 10.1109/TVCG.2016.2518158]
Wisessing P, Zibrek K, Cunningham D W, Dingliana J and McDonnell R. 2020. Enlighten me: importance of brightness and shadow for character emotion and appeal. ACM Transactions on Graphics, 39(3): #19 [DOI: 10.1145/3383195]
Wojdel A and Rothkrantz L J M. 2005. Parametric generation of facial expressions based on FACS. Computer Graphics Forum, 24(4): 743-757 [DOI: 10.1111/j.1467-8659.2005.00899.x]
Zhang K L, Liu T T, Liu Z, Zhuang Y and Chai Y J. 2020. Multimodal human-computer interactive technology for emotion regulation. Journal of Image and Graphics, 25(11): 2451-2464
张凯乐, 刘婷婷, 刘箴, 庄寅, 柴艳杰. 2020. 面向情绪调节的多模态人机交互技术. 中国图象图形学报, 25(11): 2451-2464) [DOI: 10.11834/jig.200251]
Zhang Y, Zhang L and Hossain M A. 2015. Adaptive 3D facial action intensity estimation and emotion recognition. Expert Systems with Applications, 42(3): 1446-1464 [DOI: 10.1016/j.eswa.2014.08.042]
Zibrek K, Martin S and McDonnell R. 2019. Is photorealism important for perception of expressive virtual humans in virtual reality? ACM Transactions on Applied Perception, 16(3): 1-20 [DOI:10.1145/3349609]
相关作者
相关机构
京公网安备11010802024621