Objective

2

Posed and spontaneous expression distinction is a major problem in the field of facial expression analysis. Posed expressions are deliberately performed to confuse or cheat others

while spontaneous expressions occur naturally. The difference between the posed and spontaneous delivery of the same expression by a person is little due to the subjective fraud of posed expressions. At the same time

posed and spontaneous expression distinction suffers from the problem of high intraclass differences caused by individual differences. These limitations bring difficulties in posed and spontaneous expression distinction. However

behavioral studies have shown that significant differences exist between posed and spontaneous expressions in spatial patterns. For example

compared with spontaneous smiles

the contraction of zygomatic muscles is more likely to be asymmetric in posed smiles. Moreover

constricted orbicularis oculi muscle is presented in spontaneous smiles but absent in posed smiles. Such inherent spatial patterns in posed and spontaneous expressions can be utilized to facilitate posed and spontaneous expression distinction. Therefore

modeling spatial patterns inherent in facial behavior and extracting subject-independent facial features are important for posed and spontaneous expression distinction. Previous works typically focused on spatial pattern modeling in the facial behavior. Researchers commonly use landmarks to describe motion patterns of facial muscles approximately and capture spatial patterns inherent in facial behavior based on landmark information due to the difficulty in obtaining motion patterns of facial muscles. According to the difference in modeling spatial patterns inherent in facial behavior

studies on posed and spontaneous expression distinction can be categorized into two approaches

namely

feature- and probabilistic graphical model (PGM)-based methods. Feature-based methods implicitly capture spatial patterns using handcrafted low-level or deep features extracted by deep convolution networks. However

handcrafted low-level features have difficulty in describing complex spatial patterns inherent in the facial behavior. PGM-based methods model the distribution among landmarks and explicitly capture spatial patterns existing in facial behavior using PGMs. However

PGMs frequently simplify reasoning and calculation of models through independence or energy distribution assumptions

which are sometimes inconsistent with the ground truth distribution. At the same time

PGM-based methods typically use handcrafted low-level features and thus face similar defects. An adversarial network for posed and spontaneous expression distinction is proposed to solve the problems.

Method

2

On the one hand

we use landmark displacements between onset and corresponding apex frames to describe motion patterns of facial muscles approximately and capture spatial patterns inherent in facial behavior explicitly by modeling the joint distribution between expressions and landmark displacements. On the other hand

we alleviate the problem of high intraclass differences by extracting subject-independent features. Specifically

the proposed adversarial network consists of a feature extractor

a multitask learner

a multitask discriminator

and a feature discriminator. The feature extractor attempts to extract facial features

which are discriminative for posed and spontaneous expression distinction and robust for subjects. The multitask learner is used to classify posed and spontaneous expressions as well as predict facial landmark displacement simultaneously. The multitask discriminator distinguishes the predicted expression and landmark displacement from ground truth ones. The feature discriminator is a subject classifier that can be used to measure the correlation and independence between extracted facial features and subject identities. The feature extractor is trained cooperatively with the multitask learner but in an adversarial way with the feature discriminator. Thus

the feature extractor can learn good facial features for expression distinction and landmark displacement regression but not for subject recognition. The multitask learner competes with the multitask discriminator. The distribution of predicted expression and landmark displacement converges to the distribution of ground truth labels through adversarial learning. Thus

spatial patterns can be thoroughly explored for posed and spontaneous expression distinction.

Result

2

Experimental results on three benchmark datasets

i.e. MMI(M&M Initiative)

NVIE(Natural visible and infrared facial expression)

and BioVid(Biopotential and Video) demonstrate that the proposed adversarial network can not only effectively learn subject-independent and expression-discriminative facial features

so as to improve the generalization ability of the model on unseen subjects

but also make full use of spatial and temporal patterns inherent in facial behaviors to improve the performance of posed and spontaneous expression distinction

leading to the superior performance compared to state of the arts.

Conclusion

2

Experiments demonstrate the effectiveness of the proposed method.