浏览全部资源
扫码关注微信
纸质出版日期： 2020-11-16 ，
录用日期： 2020-10-02
移动端阅览
疏颖, 毛龙彪, 陈思, 严严. 结合自监督学习和生成对抗网络的小样本人脸属性识别[J]. 中国图象图形学报, 2020,25(11):2391-2403.
Ying Shu, Longbiao Mao, Si Chen, Yan Yan. Self-supervised learning and generative adversarial network-based facial attribute recognition with small sample size training[J]. Journal of Image and Graphics, 2020,25(11):2391-2403.
目的
2
人脸属性识别是计算机视觉和情感感知等领域一个重要的研究课题。随着深度学习的不断发展，人脸属性识别取得了巨大的进步。目前基于深度学习的人脸属性识别方法大多依赖于包含完整属性标签信息的大规模数据集。然而，对于小样本数据集的属性标签缺失问题，人脸属性识别方法的准确率依然较低。针对上述问题，本文提出了一种结合自监督学习和生成对抗网络的方法来提高在小样本数据集上的人脸属性识别准确率。
方法
2
使用基于旋转的自监督学习技术进行预训练得到初始的属性识别网络；使用基于注意力机制的生成对抗网络得到人脸属性合成模型，对人脸图像进行属性编辑从而扩充训练数据集；使用扩充后的训练数据集对属性识别网络进行训练得到最终模型。
结果
2
本文在小样本数据集UMD-AED（University of Maryland attribute evaluation dataset）上进行了实验并与传统的有监督学习方法进行了比较。传统的有监督学习方法达到了63.24%的平均准确率，而所提方法达到了69.01%的平均准确率，提高了5.77%。同时，本文在CelebA（CelebFaces attributes dataset）、LFWA（labeled faces in the wild attributes dataset）和UMD-AED数据集上进行了使用自监督学习和未使用自监督学习的对比实验，验证了自监督学习在小样本数据集上的有效性。
结论
2
本文所提出的结合自监督学习和生成对抗网络的人脸属性识别方法有效提高了小样本数据集上属性识别的准确率。
Objective
2
Facial attribute recognition is an important research topic in the fields of computer vision and emotion sensing. Face
an important biological feature of human beings
contains a large number of attributes
such as expression
age
and gender. Facial attribute recognition aims to predict the different attributes in a given facial image. Facial attribute recognition has progressed considerably with the remarkable development of deep learning. State-of-the-art deep learning-based facial attribute recognition methods typically rely on large-scale training facial data with complete attribute labels. However
the number of training facial data may be limited in some real-world applications and several attribute labels of the facial image are unavailable
mainly because attribute labeling is a time-consuming and labor-intensive task. Notably
defining a standard criterion for attribute labeling is difficult for some subjective attributes. As a result
the accuracy of these methods is poor when addressing the problem of missing attribute labels in small sample size training. Previous methods attempted to find samples that match the required label from the unlabeled dataset and then added these samples to the corresponding category of the training set to augment the training data. Note that the unlabeled dataset is typically of low quality
thereby affecting the final performance of the model. Furthermore
the selection of matching samples is time consuming. Some methods directly take advantage of similar data to augment the original dataset. However
deciding whether two datasets are similar and finding similar datasets are still challenging. Current methods need further investigation on facial attribute recognition under small sample size training. A self-supervised learning and generative adversarial network (GAN)-based method is proposed in this study to solve the above-mentioned problems and improve the accuracy of facial attribute recognition for small sample size training with missing attribute labels.
Method
2
First
we adopt a rotation-based self-supervised learning technique to pretrain the attribute classification network. We use ResNet50 as the basic model of the network and modify output nodes of the last fully connected layer to four to predict the rotation angle of the input image (including 0°
90°
180° and 270°). Second
we concatenate rotated and original images along the channel dimension and use them as the input of the self-supervised learning network. Third
we utilize an attention mechanism-based GAN as the facial attribute synthesis model
where facial attributes can be edited to augment both attribute labels and training data. Specifically
the feature map is passed through a 1×1 convolution layer and then multiplied with its own transpose. In this way
we obtain attention features by multiplying the attention and feature maps. Fourth
we use this model to edit facial attributes to augment labels and training data. Finally
we use the augmented training data to train the attribute classification network initialized with self-supervised learning. We use the stochastic gradient descent algorithm during the training process. We select attributes of "baldness"
"bangs"
"black hair"
"blonde"
"brown hair"
"dense eyebrows"
"wearing glasses"
"male"
"slightly open mouth"
"mustache"
"no beard"
"pale skin"
and "young" in the experiment of synthesizing facial attributes. Both the encoder and decoder of the GAN generator contain five layers
and the discriminator also consists of five layers. We set the batch size to 64 and use a learning rate of 0.000 2.
Result
2
We use one-tenth of the data for training using CelebFaces attributes dataset (CelebA)
labeled faces in the wild attributes dataset (LFWA)
and University of Maryland attribute evaluation dataset (UMD-AED) in the experiments. The accuracy of the proposed method using CelebA and LFWA is improved by 2.42% and 3.17% respectively
compared with the traditional supervised learning-based method. The accuracy of the proposed method using UMD-AED is improved by 5.77%. We also conduct experiments on different sizes of training sets using CelebA
LFWA
and UMD-AED to verify the effectiveness of the self-supervised learning technique on the small dataset further. Experimental results showed that the model demonstrates a significantly improved performance with self-supervised learning when the size of the training set is small (from the complete training data to one-tenth of the training data). The performance of the supervised model using CelebA decreases from 90.86% to 81.97%
while the performance of the self-supervised model decreases from 90.72% to 83.57%. The performance of the supervised model using LFWA decreases by 6.11%
while the performance of the self-supervised model decreases by 3.90%. The performance of the supervised model using UMD-AED decreases by 16.95%
while the performance of the self-supervised model decreases by 11.50%.
Conclusion
2
The proposed method utilizes self-supervised learning to pretrain the initial model and uses GAN for data augmentation. Experimental results showed that our proposed method effectively improves the accuracy of facial attribute recognition for small sample size training with missing attribute labels.
人脸属性识别自监督学习生成对抗网络(GAN)数据增强小样本训练
facial attribute recognitionself-supervised learninggenerative adversarial network(GAN)data augmentationsmall sample size training
Arjovsky M, Chintala S and Bottou L. 2017. Wasserstein GAN[EB/OL].[2020-06-13].https://arxiv.org/pdf/1701.07875.pdfhttps://arxiv.org/pdf/1701.07875.pdf
Bourdev L, Maji S and Malik J. 2011. Describing people: a poselet-based approach to attribute classification//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona: IEEE: 1543-1550[DOI: 10.1109/ICCV.2011.6126413http://dx.doi.org/10.1109/ICCV.2011.6126413]
Brock A, Lim T, Ritchie J M and Weston N. 2017. SMASH: one-shot model architecture search through hypernetworks[EB/OL].[2020-06-13].https://arxiv.org/pdf/1708.05344.pdfhttps://arxiv.org/pdf/1708.05344.pdf
Cao J J, Li Y M and Zhang Z F. 2018. Partially shared multi-task convolutional neural network with local constraint for face attribute learning//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE: 4290-4299[DOI: 10.1109/CVPR.2018.00451http://dx.doi.org/10.1109/CVPR.2018.00451]
Chen B C, Chen C S and Hsu W H. 2015. Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Transactions on Multimedia, 17(6):804-815[DOI:10.1109/TMM.2015.2420374]
Deng J, Russakovsky O, Krause J, Bernstein M S, Berg A and Li F F. 2014. Scalable multi-label annotation//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Toronto: ACM: 3099-3102[DOI: 10.1145/2556288.2557011http://dx.doi.org/10.1145/2556288.2557011]
Durand T, Mehrasa N and Mori G. 2019. Learning a deep convnet for multi-label classification with partial labels//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE: 647-657[DOI: 10.1109/CVPR.2019.00074http://dx.doi.org/10.1109/CVPR.2019.00074]
Ehrlich M, Shields T J, Almaev T and Amer M R. 2016. Facial attributes classification using multi-task representation learning//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Las Vegas: IEEE: 47-55[DOI: 10.1109/CVPRW.2016.99http://dx.doi.org/10.1109/CVPRW.2016.99]
Eidinger E, Enbar R and Hassner T. 2014. Age and gender estimation of unfiltered faces. IEEE Transactions on Information Forensics and Security, 9(12):2170-2179[DOI:10.1109/TIFS.2014.2359646]
Finn C, Xu K and Levine S. 2018. Probabilistic model-agnostic meta-learning//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal: Curran Associates Inc., 9516-9527
Gidaris S, Singh P and Komodakis N. 2018. Unsupervised representation learning by predicting image rotations[EB/OL].[2020-06-03].https://arxiv.org/pdf/1803.07728.pdfhttps://arxiv.org/pdf/1803.07728.pdf
Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: ACM: 2672-2680
Grant E, Finn C, Levine S, Darrell T and Griffiths T. 2018. Recasting gradient-based meta-learning as hierarchical Bayes[EB/OL].[2020-06-13].https://arxiv.org/pdf/1801.08930.pdfhttps://arxiv.org/pdf/1801.08930.pdf
Han H, Jain A K, Wang F, Shan S G and Chen X L. 2017. Heterogeneous face attribute estimation:a deep multi-task learning approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(11):2597-2609[DOI:10.1109/TPAMI.2017.2738004]
Hand E M, Castillo C and Chellappa R. 2018. Doing the best we can with what we have: multi-label balancing with selective learning for attribute prediction//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans: AAAI: 1-9
He K K, Fu Y W, Zhang W H, Wang C J, Jiang Y G, Huang F T and Xue X Y. 2018. Harnessing synthesized abstraction images to improve facial attribute recognition//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm: ACM: 733-740
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 770-778[DOI: 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90]
He Z L, Zuo W M, Kan M N, Shan S G and Chen X L. 2019. AttGAN:facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 28(11):5464-5478[DOI:10.1109/TIP.2019.2916751]
Huang S Y, Li X, Cheng Z Q, Zhang Z F and Hauptmann A. 2018. GNAS: a greedy neural architecture search method for multi-attribute learning//Proceedings of the 26th ACM International Conference on Multimedia. Seoul: ACM: 2049-2057[DOI: 10.1145/3240508.3240588http://dx.doi.org/10.1145/3240508.3240588]
Ioffe S and Szegedy C. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift[EB/OL].[2020-06-13].https://arxiv.org/pdf/1502.03167.pdfhttps://arxiv.org/pdf/1502.03167.pdf
Karras T, Laine S and Aila T. 2019. A style-based generator architecture for generative adversarial networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE: 4396-4405[DOI: 10.1109/CVPR.2019.00453http://dx.doi.org/10.1109/CVPR.2019.00453]
Kim B K, Roh J, Dong S Y and Lee S Y. 2016. Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. Journal on Multimodal User Interfaces, 10(2):173-189[DOI:10.1007/s12193-015-0209-0]
Kumar N, Belhumeur P N and Nayar S. 2008. Facetracer: a search engine for large collections of images with faces//Proceedings of the 10th European Conference on Computer Vision. Marseille: Springer: 340-353[DOI: 10.1007/978-3-540-88693-8_25http://dx.doi.org/10.1007/978-3-540-88693-8_25]
Kumar N, Berg A, Belhumeur P N and Nayar S. 2011. Describable visual attributes for face verification and image search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10):1962-1977[DOI:10.1109/TPAMI.2011.48]
LeCun Y, Bottou L, Bengio Y and Haffner P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324[DOI:10.1109/5.726791]
Li H J and Suen C Y. 2016. Robust face recognition based on dynamic rank representation. Pattern Recognition, 60:13-24[DOI:10.1016/j.patcog.2016.05.014]
Liu Z W, Luo P, Wang X G and Tang X O. 2015. Deep learning face attributes in the wild//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago: IEEE: 3730-3738[DOI: 10.1109/ICCV.2015.425http://dx.doi.org/10.1109/ICCV.2015.425]
Mao L B, Yan Y, Xue J H and Wang H Z. 2020. Deep multi-task multi-label CNN for effective facial attribute classification. IEEE Transactions on Affective Computing: 1-11[DOI: 10.1109/TAFFC.2020.2969189http://dx.doi.org/10.1109/TAFFC.2020.2969189]
Marasamy P and Sumathi S. 2012. Automatic recognition and analysis of human faces and facial expression by LDA using wavelet transform//Proceedings of 2012 International Conference on Computer Communication and Informatics. Coimbatore: IEEE: 1-4[DOI: 10.1109/ICCCI.2012.6158798http://dx.doi.org/10.1109/ICCCI.2012.6158798]
Odena A, Olah C and Shlens J. 2017. Conditional image synthesis with auxiliary classifier GANs//Proceedings of the 34th International Conference on Machine Learning. Sydney: ACM: 2642-2651
Pfister T, Charles J and Zisserman A. 2014. Domain-adaptive discriminative one-shot learning of gestures//Proceedings of the 13th European Conference on Computer Vision. Zurich: Springer: 814-829[DOI: 10.1007/978-3-319-10599-4_52http://dx.doi.org/10.1007/978-3-319-10599-4_52]
Radford A, Metz L and Chintala S. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL].[2020-06-13].https://arxiv.org/pdf/1511.06434.pdfhttps://arxiv.org/pdf/1511.06434.pdf
Sun C, Shrivastava A, Singh S and Gupta A. 2017. Revisiting unreasonable effectiveness of data in deep learning era//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 843-852[DOI: 10.1109/ICCV.2017.97http://dx.doi.org/10.1109/ICCV.2017.97]
Sun Y, Wang X G and Tang X O. 2015. Deeply learned face representations are sparse, selective, and robust//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE: 2892-2900[DOI: 10.1109/CVPR.2015.7298907http://dx.doi.org/10.1109/CVPR.2015.7298907]
Tsai Y H H and Salakhutdinov R. 2017. Improving one-shot learning through fusing side information[EB/OL].[2020-06-13].https://arxiv.org/pdf/1710.08347.pdfhttps://arxiv.org/pdf/1710.08347.pdf
Vahdat A. 2017. Toward robustness against label noise in training deep discriminative neural networks//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: ACM: 5596-5605
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł and Polosukhin I. 2017. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 5998-6008
Wang Y Q, Yao Q M, Kwok J and Ni L M. 2019. Generalizing from a few examples: a survey on few-shot learning[EB/OL].[2020-06-13].https://arxiv.org/pdf/1904.05046.pdfhttps://arxiv.org/pdf/1904.05046.pdf
Xu M, Jin R and Zhou Z H. 2013. Speedup matrix completion with side information: application to multi-label learning//Proceedings of the 27th Annual Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc., 2301-2309
Yu Y F, Dai D Q, Ren C X and Huang K K. 2017. Discriminative multi-layer illumination-robust feature extraction for face recognition. Pattern Recognition, 67:201-212[DOI:10.1016/j.patcog.2017.02.004]
Zhang H, Goodfellow I, Metaxas D and Odena A. 2018a. Self-attention generative adversarial networks[EB/OL].[2020-06-13].https://arxiv.org/pdf/1805.08318.pdfhttps://arxiv.org/pdf/1805.08318.pdf
Zhang G, Kan M N, Shan S G and Chen X L. 2018b. Generative adversarial network with spatial attention for face attribute editing//Proceedings of the 15th European Conference on Computer Vision. Munich: Springer: 422-437[DOI: 10.1007/978-3-030-01231-1_26http://dx.doi.org/10.1007/978-3-030-01231-1_26]
Zhong Y, Sullivan J and Li H B. 2016. Leveraging mid-level deep representations for predicting face attributes in the wild//Proceedings of 2016 IEEE International Conference on Image Processing. Phoenix: IEEE: 3239-3243[DOI: 10.1109/ICIP.2016.7532958http://dx.doi.org/10.1109/ICIP.2016.7532958]
Zhuang N, Yan Y, Chen S, Wang H Z and Shen C H. 2018. Multi-label learning based deep transfer neural network for facial attribute classification. Pattern Recognition, 80:225-240[DOI:10.1016/j.patcog.2018.03.018]
相关作者
相关机构
京公网安备11010802024621