Objective

2

Facial expression is an effective means of communicating inner feelings and intentions. With the rapid development of artificial intelligence

facial expression recognition has become a critical part of human-computer interaction today. Research on human facial expression recognition technology bears important theoretical significance and practical application value. A video sequence contains more emotional information than a static expression image. In recent years

facial expression recognition based on video sequences has gradually become a field of interest in computer vision because expressing emotions is a dynamic process. Information obtained from a single-frame image is not as rich as that from a video sequence

and the accuracy of the former is low compared with the latter. Presently

two mainstream solutions for video facial expression recognition are available. The advantage of motion or movement direction of the face unit to analyze facial expressions is that it does not require selection of video frames. Rather

this method directly extracts the dynamic feature of the video for processing. However

the recognition process is complex

and the recognition rate is low. Another method is basing on the image of facial expressions to identify the expression category. Despite its high recognition rate

this method requires processing the original video sequence in advance. In a complete sequence of expressions

obvious expression plays a key role in feature extraction and recognition

but a video sequence also presents a neutral expression

which may interfere training of the model parameters and affect the output. Therefore

we need to manually select noticeable features of the expression sequence from the original video sequence. However

this operation generates extra work and affects the accuracy of experiment. This study proposes a new modified method of dynamic time warping

which is called the sliding window dynamic time warping (SWDTW)

to automatically select the distinct facial expressions in a video sequence. The method can reduce redundant input information and improve adaptability during experiment. Moreover

noise in feature extraction and expression recognition is lessened. In video facial expression recognition

identification results are greatly influenced by environmental lights

and traditional feature extraction requires excessive manual intervention. Therefore

this study proposes a facial expression recognition method based on deep convolution neural network

which is a new type of neural network. This network combines traditional artificial neural network with deep learning technology

and it has achieved considerable success in the area of image processing. Convolution neural network has two main characteristics. The first is its strategy in establishing local connection between neurons

and the second is the sharing of neurons in the same layer. These characteristics reduce the complexity of the model and the number of parameters to be trained. This network structure can achieve several degrees of invariance

such as translation

scale

and deformation.

Method

2

First

this method uses an algorithm to intercept front face frame in an expression sequence after a series of normalized processing by indicating the gradient direction histogram feature to calculate cost matrix and adding the sliding window mechanism to the cost matrix. Second

the average distances of all sliding windows are calculated

and the global optimal selection of the expression sequence is obtained by intercepting the sequence corresponding to the minimum distance. Finally

theoretical analysis and experimental verification are performed to determine the structure and parameters of the convolution neural network. In this work

the Alexnet network is selected as a reference because it won in the image classification competition of ImageNet Large Scale Visual Recognition Challenge (ⅡSVRC) in 2012. In view of the feature of facial expression images

this study makes certain adjustments on the original AlexNet network to better meet facial expression recognition and improves the network's utilization of overall information by reducing two convolution layers and adding a pooling layer. During convolution

the ReLU activation function is used to replace the traditional sigmoid and tanh activation function

increase the training speed of the model

and solve gradient disappearance. Dropout technology to address network over-fitting problem is also introduced. Finally

this study uses two fully connected layers to classify facial expression. Regularized facial expression video sequence is used for unsupervised learning and facial expression classification through deep convolution neural network. Images in video sequence undergo classification probability to determine which belong to each expression category

and the final identification results of the video sequence are obtained.

Result

2

Five cross-validation experiments are conducted on CK+ and MMI database. This method performs better than the randomly selected video sequence and manual feature extraction methods in terms of recognition and generalization. In CK+ and MMI database

the average recognition accuracies are 92.54% and 74.67%

respectively

which are 19.86% and 22.24% higher than those of randomly selected video sequences. In addition

in comparison with other methods

SWDTW achieves better recognition performance.

Conclusion

2

The proposed method exhibits good performance and adaptability during preprocessing

feature extraction

and recognition of facial expression system. SWDTW effectively achieves the selection of expression sequence

and the designed convolution neural network improves the robustness of facial expression classification based on video.