In view of existing algorithms

volume local binary pattern is applied to the feature extraction of video frames. However

problems such as large feature dimension

weak robustness to illumination

and noise exist. This study proposes a new feature description algorithm

which is temporal-spatial local ternary pattern moment. This algorithm introduces three value patterns

and it is extended to the temporal-spatial series to describe the variety of pixel values among adjacent frames. The value of texture feature is represented by the energy values of the three value model matrixes

which are calculated according to the gray-level co-occurrence matrix. Considering that the temporal-spatial local ternary pattern moment only describes the texture feature

it lacks the expression of image edge and direction information. Therefore

it cannot fully describe the characteristics of emotional videos. The feature of 3D histograms of oriented gradients is further fused to enhance the description of the emotion feature. Composite spatio-temporal features are obtained by combining two different features. First

the emotional videos are preprocessed

and five frame images are obtained by K mean clustering

which are used as the expression and body posture emotion sequences. Second

TSLTPM and 3DHOG features are extracted from the expression and gesture emotion sequences

and the minimum Euclidean distance of the feature between the test sequence and labeled emotion training set is calculated. The calculated value is used as independent evidence to construct the basic probability assignment function. Finally

according to the rules of D-S evidence theory

the expression recognition result is obtained by fused BPA. Experimental results on the bimodal expression and body posture emotion database show that complex spatio-temporal features exhibit good recognition performance. The average recognition rates of 83.06% and 94.78% are obtained in the single model identification of facial expressions and gestures

respectively

compared with other algorithms. The average recognition rate of the single-expression model is 9.27%

12.89%

1.87%

and 1.13% higher than those of VLBP

LBP-TOP

TSLTPM

and 3DHOG

respectively. The average recognition rate of the single-gesture model is 24.61%

27.55%

1.18%

and 0.98% higher than those of VLBP

LBP-TOP

TSLTPM

and 3DHOG

respectively. The average recognition rate after the fusion of these two models is 96.86%

which is higher than the rate obtained by a single model. This result confirms the effectiveness of emotion recognition under the fusion of expression and gesture. The TSLTPM feature proposed in our paper extends the VLBP

which is effective in describing the local features of video images

into the temporal–spatial local ternary pattern. The proposed feature has low dimensionality

and it can enhance the robustness to illumination and noise. The composite spatio-temporal features fused with 3DHOG and TSLTPM can fully describe the effective information of emotional videos

and it enhances the classification performance of such videos. The effectiveness of the proposed algorithm in comparison with other typical feature extraction algorithms is also demonstrated. The proposed algorithm is proven suitable for identifying the emotion of static background videos

and the superiority of the fusion method in this study is verified.