Expression recognition algorithm for parallel convolutional neural networks

Linlin Xu; Shumei Zhang; Junli Zhao

doi:10.11834/jig.180346

Image Analysis and Recognition | Views : 0 下载量: 68 CSCD: 21

PDF
Export
Share
Collection
Album

Expression recognition algorithm for parallel convolutional neural networks
Vol. 24, Issue 2, Pages: 227-236(2019)
Received：04 June 2018，

Revised：2018-8-16，

Published：16 February 2019
DOI： 10.11834/jig.180346
稿件说明：

移动端阅览

DOI：

Linlin Xu, Shumei Zhang, Junli Zhao. Expression recognition algorithm for parallel convolutional neural networks[J]. Journal of Image and Graphics, 2019, 24(2): 227-236. DOI： 10.11834/jig.180346.

摘要

目的

表情识别在商业、安全、医学等领域有着广泛的应用前景，能够快速准确地识别出面部表情对其研究与应用具有重要意义。传统的机器学习方法需要手工提取特征且准确率难以保证。近年来，卷积神经网络因其良好的自学习和泛化能力得到广泛应用，但还存在表情特征提取困难、网络训练时间过长等问题，针对以上问题，提出一种基于并行卷积神经网络的表情识别方法。

方法

首先对面部表情图像进行人脸定位、灰度统一以及角度调整等预处理，去除了复杂的背景、光照、角度等影响，得到了精确的人脸部分。然后针对表情图像设计一个具有两个并行卷积池化单元的卷积神经网络，可以提取细微的表情部分。该并行结构具有3个不同的通道，分别提取不同的图像特征并进行融合，最后送入SoftMax层进行分类。

结果

实验使用提出的并行卷积神经网络在CK+、FER2013两个表情数据集上进行了10倍交叉验证，最终的结果取10次验证的平均值，在CK+及FER2013上取得了94.03%与65.6%的准确率。迭代一次的时间分别为0.185 s和0.101 s。

结论

为卷积神经网络的设计提供了一种新思路，可以在控制深度的同时扩展广度，提取更多的表情特征。实验结果表明，针对数量、分辨率、大小等差异较大的表情数据集，该网络模型均能够获得较高的识别率并缩短训练时间。

Abstract

Objective

Face emotion recognition is widely applied in the fields of commercial

security

and medicine. Rapid and accurate identification of facial expressions are of great significance for their research and application. Several traditional machine learning methods

such as support vector machine (SVM)

principal component analysis (PCA)

and local binary pattern (LBP) are used to identify facial expressions. However

these traditional machine learning algorithms require manual feature extraction. In this process

some features are hidden or deliberately enlarged due to many human interventions

which affect accuracy. In recent years

convolutional neural networks (CNNs) have been used extensively in image recognition due to their good self-learning and generalization capabilities. However

several problems

such as difficulty in facial expression feature extraction and long training time of neural network

are still observed with neural network training. This study presents an expression recognition method based on parallel CNN to solve the aforementioned problems.

Method

First

a series of preprocessing operations is performed on facial expression images. For example

an original image is detected by using an AdaBoost cascade classifier to remove the complex background and obtain the face part. Then

a face image is compensated by illumination

a histogram equalization method is used to stretch the image nonlinearly

and the pixel value of the image is reallocated. Finally

affine transformation is used to achieve face alignment. The preceding preprocessing can remove complex background effects

compensate lighting

and adjust the angle to obtain more accurate face parts than that of the original image. Then

a CNN with two parallel convolution and pooling structures

which can extract subtle expressions

is designed for facial expression images. This parallel unit is the core unit of the CNN and comprises a convolutional layer

a pooling layer

and an activation function ReLu. This parallel structure has three different channels

in which each channel has different number of convolutions

pooling layers

and ReLu to extract different image features and fuse the extracted features. The second parallel processing unit can perform convolution and pooling on the extracted features by the first parallel processing unit and reduce the dimension of the image and shorten the training time of CNN. Finally

the previously merged features are sent to the SoftMax layer for expression classification.

Result

CK+ and FER2013 expression datasets that have undergone pre-processing and data enhancement are divided into 10 equal parts. Then

training and testing are performed on 10 parts

and the final accuracy is the average of the 10 results. Experimental results show that the accuracy increases and time decreases remarkably compared with traditional machine learning methods

such as SVM

PCA

and LBP or their combination and other classical CNNs

such as AlexNet and GoogLeNet. Finally

CK+ and FER2013 achieve 94.03% and 65.6% accuracy

and the iteration time reaches 0.185 s and 0.101 s

respectively.

Conclusion

This study presents a new parallel CNN structure that extracts the features of facial expressions by using three different convolutional and pooling structures. The three paths have different combinations of convolutional and pooling layers

and they can extract different image features. The different extracted features are combined and sent to the next layer for processing. This study provides a new concept for the design of CNNs

which can extend the breadth of CNN and control the depth. The proposed CNN can extract many expressions that are ignored or difficult to extract. CK+ and FER2013 expression datasets have large difference in quantity

size

and resolution. The experiments of CK+ and FER2013 show that the model can extract the precise and subtle features of facial expression images in a relatively short time under the premise of ensuring the recognition rate.

关键词

Keywords

references

Anderson K, McOwan P W. A real-time automated system for the recognition of human facial expressions[J]. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics), 2006, 36(1):96-105.[DOI:10.1109/TSMCB.2005.854502

Pramerdorfer C, Kampel M. Facial expression recognition using convolutional neural networks: state of the art[EB/OL].[2018-05-20] . https://arxiv.org/pdf/1612.02903.pdf https://arxiv.org/pdf/1612.02903.pdf

Shan C F, Gong S G, McOwan P W. Facial expression recognition based on Local Binary Patterns:A comprehensive study[J]. Image and Vision Computing, 2009, 27(6):803-816.[DOI:10.1016/j.imavis.2008.08.005

Berretti S, Ben Amor B, Daoudi M, et al. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints[J]. The Visual Computer, 2011, 27(11):1021-1036.[DOI:10.1007/s00371-011-0611-x

Albiol A, Monzo D, Martin A, et al. Face recognition using HOG-EBGM[J]. Pattern Recognition Letters, 2008, 29(10):1537-1543.[DOI:10.1016/j.patrec.2008.03.017

Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE, 2015: 1-9.[ DOI: 10.1109/CVPR.2015.7298594 http://dx.doi.org/10.1109/CVPR.2015.7298594 ]

Burkert P, Trier F, Afzal M Z, et al. DeXpression: deep convolutional neural network for expression recognition[EB/OL].[2018-05-20] . https://arxiv.org/pdf/1509.05371.pdf https://arxiv.org/pdf/1509.05371.pdf

Yang G L, Deng X J, Liu C. Facial expression recognition model based on deep spatiotemporal convolutional neural networks[J]. Journal of Central South University:Science and Technology, 2016, 47(7):2311-2319.

杨格兰, 邓晓军, 刘琮.基于深度时空域卷积神经网络的表情识别模型[J].中南大学学报:自然科学版, 2016, 47(7):2311-2319. [DOI:10.11817/j.issn.1672-7207.2016.07.037

Lecun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 2015, 521(7553):436-444.[DOI:10.1038/nature14539

Zeiler M D, Fergus R. Stochastic pooling for regularization of deep convolutional neural networks[EB/OL].[2018-05-20] . https://arxiv.org/pdf/1301.3557.pdf https://arxiv.org/pdf/1301.3557.pdf

Nair V, Hinton G E. Rectified linear units improve restricted boltzmann machines[C]//Proceedings of the 27th International Conference on International Conference on Machine Learning. Haifa, Israel: Omnipress, 2010: 807-814. http://120.52.51.17/www.cs.toronto.edu/~fritz/absps/reluICML.pdf .

Wei Z. Research and implementation of face recognition based on deep learning based on Caffe platform[D]. Xi'an: Xi'an University of Electronic Science and Technology, 2015. http://cdmd.cnki.com.cn/Article/CDMD-10701-1016245785.htm .

魏正.基于Caffe平台深度学习的人脸识别研究与实现[D].西安: 西安电子科技大学, 2015.

Lucey P, Cohn J F, Kanade T, et al. The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression[C]//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. San Francisco, CA: IEEE, 2010: 94-101.[ DOI: 10.1109/CVPRW.2010.5543262 http://dx.doi.org/10.1109/CVPRW.2010.5543262 ]

Kotsia I, Pitas I. Facial expression recognition in image sequences using geometric deformation features and support vector machines[J]. IEEE Transactions on Image Processing, 2007, 16(1):172-187.[DOI:10.1109/TIP.2006.884954

Hinton G E, Srivastava N, Krizhevsky A, et al. Improving neural networks by preventing co-adaptation of feature detectors[EB/OL].[2018-05-20] . https://arxiv.org/pdf/1207.0580.pdf https://arxiv.org/pdf/1207.0580.pdf

Fu Q M, Liu Q, Wang H, et al. A novel off policy Q(λ) algorithm based on linear function approximation[J]. Chinese Journal of Computers, 2014, 37(3):677-686.

傅启明, 刘全, 王辉, 等.一种基于线性函数逼近的离策略Q(λ)算法[J].计算机学报, 2014, 37(3):677-686. [DOI:10.3724/SP.J.1016.2013.00677

He J, Cai J F, Fang L Z, et al. Facial expression recognition based on LBP/VAR and DBN model[J]. Application Research of Computers, 2016, 33(8):2509-2513.

何俊, 蔡建峰, 房灵芝, 等.基于LBP/VAR与DBN模型的人脸表情识别[J].计算机应用研究, 2016, 33(8):2509-2513. [DOI:10.3969/j.issn.1001-3695.2016.08.060

Zhang B. Facial expression recognition based on Gabor and conditional random field[D]. Jinan: Shan Dong University, 2015. http://cdmd.cnki.com.cn/Article/CDMD-10422-1015372484.htm .

张博.基于Gabor和条件随机场的人脸表情识别[D].济南: 山东大学, 2015.

Goodfellow I J, Erhan D, Carrier P L, et al. Challenges in representation learning:A report on three machine learning contests[J]. Neural Networks, 2015, 64:59-63.[DOI:10.1016/j.neunet.2014.09.005

Alert me when the article has been cited

提交

Uncertainty domain awareness network for cross-domain few-shot image classification

Double-pooling residual classification network based on feature reordering attention mechanism

Data-free knowledge distillation for target class classification

Comprehensive review of methods for vehicle logo recognition in intelligent transportation systems

TransAS-UNet： regional segmentation of breast cancer Swin Transformer and of UNet algorithm