掩膜融合下的人脸图像质量评价方法

李雷达; 殷杨涛; 吴金建; 董伟生; 石光明

doi:10.11834/jig.210986

图像分析和识别 | 浏览量 : 0 下载量: 199 CSCD: 1

PDF
导出
分享
收藏
专辑

掩膜融合下的人脸图像质量评价方法
Mask-fused human face image quality assessment method
2022年27卷第12期页码：3476-3490
收稿：2021-10-14，

修回：2021-12-10，

录用：2021-12-17，

纸质出版：2022-12-16
DOI： 10.11834/jig.210986
稿件说明：

移动端阅览

李雷达, 殷杨涛, 吴金建, 董伟生, 石光明. 掩膜融合下的人脸图像质量评价方法[J]. 中国图象图形学报, 2022,27(12):3476-3490. DOI： 10.11834/jig.210986.

Leida Li, Yangtao Yin, Jinjian Wu, Weisheng Dong, Guangming Shi. Mask-fused human face image quality assessment method[J]. Journal of Image and Graphics, 2022, 27(12): 3476-3490. DOI： 10.11834/jig.210986.

摘要

目的

人脸识别技术已经在众多领域中得到广泛应用，然而现有识别方法对于人脸图像的质量要求普遍较高，低质量图像会严重影响系统的识别性能，产生误判。人脸图像质量评价方法可用于高质量图像的筛选，对改善人脸识别系统的性能有重要作用。不同于传统的图像质量评价，人脸图像质量评价是一种可用性评价，目前对其研究较少。人们在进行人脸识别时往往主要通过眼睛、鼻子、嘴等关键区域；基于此，本文提出了一种基于掩膜的人脸图像质量无参考评价方法，通过挖掘脸部关键区域对人脸识别算法的影响计算人脸图像质量。

方法

人脸识别方法通常需要比较输入人脸图像和高质量基准图像之间的特征相似度；本文从另一个角度出发，在输入人脸图像的基础上构造低可用性图像作为伪参考，并通过计算输入人脸图像和伪参考图像间的相似性获得输入人脸图像的质量评价分数。具体地，对一幅输入的人脸图像，首先对其关键区域添加掩膜获得低可用性质量的掩膜人脸图像，然后将输入图像和掩膜图像输入特征提取网络以获得人脸特征，最后计算特征间的距离获得输入人脸图像的质量分数。

结果

用AOC(错误拒绝曲线围成的区域面积)作为评估指标，在5个数据集上将本文方法与其他主流的人脸质量评价方法进行了充分比较，在LFW(labeled faces in the wild)数据集中比性能第2的模型提升了14.8%，在CelebA(celebFaces attribute)数据集中提升了0.1%，在DDFace(diversified distortion face)数据集中提升了2.9%，在VGGFace2(Visual Geometry Group Face2)数据集中提升了3.7%，在CASIA-WebFace(Institute of Automation，Chinese Academy of Science-Website Face)数据集中提升了4.9%。

结论

本文提出的基于掩膜的人脸图像质量评价方法，充分利用了人脸识别的关键性区域，将人脸识别的特点融入到人脸图像质量评价算法的设计中，能够在不需要参考图像的条件下准确预测出不同失真程度下的人脸图像质量分数，并且性能优于目前的主流方法。

Abstract

Objective

Human face recognition has been developing for biometrics applications like online payment and security. Face-related recognition systems are usually deployed in an open environment in reality

which is challenged for the robustness problem. The changing external environment (e.g.

improper exposure

poor lighting

extreme weather conditions

background interference)

can intervene diversified distortions to the face images like low contrast

blurring and occlusion

which significantly degrades the performance of the face-related recognition system. Therefore

an accurate face image quality assessment method is highly required to improve the performance of the face recognition system from two perspectives as mentioned below: 1) face-related image quality model can be used to filter out low-quality face images since the performance of face recognition systems is often affected by low-quality images

thus avoiding invalid recognition and improving the recognition efficiency. 2) Traditional face recognition features can be enhanced in terms of the integrated facial quality features. In contrast to the traditional image quality assessment approaches

face-related image quality assessment can be achieved with specific face recognition algorithms only. The existing face-related image quality model scan be divided into handcrafted feature-based and deep learning-based.

Method

We develop a new mask-based method for face-related image quality assessment. From the perspective of human recognition

the quality of a face image is mainly determined by the key regions of the face image (eyes

nose

and mouth). Changes in these regions will have different impacts on the recognition performance for face-related images with multi-level quality. A mask added on these regions will also have different impacts for different face images. For example

high-quality images masked tends to have greater impact on the recognition performance compared with low-quality face images. Such a mask can be designed to cover the key regions

and the quality of a face image can be achieved by measuring the influence of the masking operation. Our human face-related image quality model can be segmented into two categories: 1) the masking operation on face images; 2) the quality score regression. Specifically

the mask is added to the key regions at first for an input face image to be evaluated. Next the face image pair is obtained containing the input image and the masked image. Finally

image pair is input into the deep feature extraction module

producing the qualified features. The objective quality score of the input face image is obtained in terms of the feature pair regression. Our method is called mask-based face image quality (MFIQ). For model training

we build a new DDFace(Diversified Distortion Face) database

which contains a total of 350 000 distorted face images of 1 000 people. We use 280 000 face images as the training set and the rest of it as the testing set. We train the model for 40 epochs with the learning rate 0.001 and batch size 32.

Result

In the experiments

five face image datasets are used

including our DDFace-built and four existing face recognition datasets like LFW(Labeled Faces in the Wild)

VGGFace2(Visual Geometry Group Face2)

CASIA-WebFace(Institute of Automation Chinese Academy of Science-Website Face) and CelebA(CelebFaces Attribute). Our proposed MFIQ model is compared with the popular deep face image quality models

including face quality net-v0 (FaceQnet-v0)

face quality net-v1(FaceQnet-v1) and stochastic embedding robustness-face image quality (SER-FIQ). Under the metric area over curve(AOC)

our model performance is improved by 14.8%

0.1%

2.9%

3.7% and 4.9% in comparison with LFW

CelebA

DDFace

VGGFace2 and CASIA-WebFace databases

respectively. Furthermore

our MFIQ model is used to predict the face-related image quality in different datasets and the quality distributions of images are calculated. The experimental results show that our distributions predicted is close to the real distributions. Our MFIQ model performance is also compared with the other three models according to face-related images evaluation from singles and multiples. The results show that the proposed MFIQ performs better than SER-FIQ

FaceQnet-v0 and FaceQnet-v1.

Conclusion

Our research potentials are focused on more robustness and distinguishing ability for the key elements of multiple-level distorted images.

关键词

Keywords

references

Abdel-Mottaleb M and Mahoor M H. 2007. Application notes-algorithms for assessing the quality of facial images. IEEE Computational Intelligence Magazine, 2(2): 10-17[DOI: 10.1109/MCI.2007.353416]

Beveridge J R, Givens G H, Phillips P J, Draper B A and Lui Y M. 2008. Focus on quality, predicting FRVT 2006 performance//Proceedings of the 8th IEEE International Conference on Automatic Face and Gesture Recognition. Amsterdam, the Netherlands: IEEE 1-8[DOI: 10.1109/AFGR.2008.4813375 http://dx.doi.org/10.1109/AFGR.2008.4813375 ]

Beveridge J R, Givens G H, Phillips P J, Draper B A, Bolme D S and Lui Y M. 2010. FRVT 2006: quo Vadis face quality. Image and Vision Computing, 28(5): 732-743[DOI: 10.1016/j.imavis.2009.09.005]

Cao Q, Shen L, Xie W D, Parkhi O M and Zisserman A. 2018. VGGFace2: a dataset for recognising faces across pose and age//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Xi'an, China: IEEE: 67-74[DOI: 10.1109/FG.2018.00020 http://dx.doi.org/10.1109/FG.2018.00020 ]

Chen J S, Deng Y, Bai G C and Su G D. 2014. Face image quality assessment based on learning to rank. IEEE Signal Processing Letters, 22(1): 90-94[DOI: 10.1109/LSP.2014.2347419]

Deng J K, Guo J, Xue N N and Zafeiriou S. 2019a. ArcFace: additive angular margin loss for deep face recognition//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 4690-4699[DOI: 10.1109/CVPR.2019.00482 http://dx.doi.org/10.1109/CVPR.2019.00482 ]

Deng J K, Guo J, Zhou Y X, Yu J K, Kotsia I and Zafeiriou S. 2019b. RetinaFace: single-stage dense face localisation in the wild[EB/OL]. [2021-11-26] . https://arxiv.org/pdf/1905.00641.pdf https://arxiv.org/pdf/1905.00641.pdf

Fang Y M,Sui X J, Yan J B, Liu X L and Huang L P. 2021. Progress in no-reference image quality assessment. Journal of Image and Graphics, 26(2): 265-286

方玉明, 眭相杰, 鄢杰斌, 刘学林, 黄丽萍. 2021. 无参考图像质量评价研究进展. 中国图象图形学报, 26(2): 265-286)[DOI: 10.11834/jig.200274]

Ferrara M, Franco A, Maio D and Maltoni D. 2012. Face image conformance to ISO/ICAO standards in machine readable travel documents. IEEE Transactions on Information Forensics and Security, 7(4): 1204-1213[DOI: 10.1109/TIFS.2012.2198643]

Fu Z Q, Shao F, Jiang G Y and Yu M. 2018. Image retargeting quality assessment via bidirectional similarity transformation. Journal of Image and Graphics, 23(4): 490-499

富振奇, 邵枫, 蒋刚毅, 郁梅. 2018. 结合双向相似性变换的重定向图像质量评价. 中国图象图形学报, 23(4): 490-499)[DOI: 10.11834/jig.170502]

Grother P and Tabassi E. 2007. Performance of biometric quality measures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(4): 531-543[DOI: 10.1109/TPAMI.2007.1019]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Hernandez-Ortega J, Galbally J, Fierrez J and Beslay L. 2020. Biometric quality: review and application to face recognition with FaceQnet[EB/OL]. [2021-11-26] . https://arxiv.org/abs/2006.03298.pdf https://arxiv.org/abs/2006.03298.pdf

Hernandez-Ortega J, Galbally J, Fierrez J, Haraksim R and Beslay L. 2019. FaceQnet: quality assessment for face recognition based on deep learning//Proceedings of 2019 International Conference on Biometrics. Crete, Greece: IEEE: 1-8[DOI: 10.1109/ICB45273.2019.8987255 http://dx.doi.org/10.1109/ICB45273.2019.8987255 ]

Huang G B, Mattar M, Berg T and Learned-Miller E. 2008. Labeled faces in the wild: a database for studying face recognition in unconstrained environments//Proceedings of the Workshop on Faces in "Real-Life" Images: Detection, Alignment, and Recognition. Marseille, France: Erik Learned-Miller and Andras Ferencz and Frédéric Jurie.

Liao P, Lin H X, Zeng P P, Bai S X, Ma H M and Ding S. 2012. Facial image quality assessment based on support vector machines//Proceedings of 2012 International Conference on Biomedical Engineering and Biotechnology. Macau, China: IEEE: 810-813[DOI: 10.1109/iCBEB.2012.221 http://dx.doi.org/10.1109/iCBEB.2012.221 ]

Liu W J, Bing X H, Jiang W T and Zhang S C. 2018. Face recognition of generalized parallel two-dimensional complex discriminant analysis. Journal of Image and Graphics, 23(9): 1359-1370

刘万军, 邴晓环, 姜文涛, 张晟翀. 2018. 广义并行2维复判别分析的人脸识别. 中国图象图形学报, 23(9): 1359-1370)[DOI: 10.11834/jig.170671]

Liu W Y, Wen Y D, Yu Z D, Li M, Raj B and Song L. 2017. SphereFace: deep hypersphere embedding for face recognition//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 6738-6746[DOI: 10.1109/CVPR.2017.713 http://dx.doi.org/10.1109/CVPR.2017.713 ]

Liu Z W, Luo P, Wang X G and Tang X O. 2015. Deep learning face attributes in the wild//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 3730-3738[DOI: 10.1109/ICCV.2015.425 http://dx.doi.org/10.1109/ICCV.2015.425 ]

Luo H T. 2004. A training-based no-reference image quality assessment algorithm//Proceedings of 2004 International Conference on Image Processing. Singapore, Singapore: IEEE: 2973-2976[DOI: 10.1109/ICIP.2004.1421737 http://dx.doi.org/10.1109/ICIP.2004.1421737 ]

Mittal A, Moorthy A K and Bovik A C. 2012a. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, 21(12): 4695-4708[DOI: 10.1109/TIP.2012.2214050]

Mittal A, Soundararajan R and Bovik A C. 2012b. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3): 209-212[DOI: 10.1109/LSP.2012.2227726]

Oliva A and Torralba A. 2001. Modeling the shape of the scene: a holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3): 145-175[DOI: 10.1023/A:1011139631724]

Ou F Z, Chen X Y, Zhang R X, Huang Y G, Li S X, Li J L, Li Y, Cao L J and Wang Y G. 2021. SDD-FIQA: unsupervised face image quality assessment with similarity distribution distance//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE: 7666-7675[DOI: 10.1109/CVPR46437.2021.00758 http://dx.doi.org/10.1109/CVPR46437.2021.00758 ]

Ou F Z, Wang Y G, Li J, Zhu G P and Kwong S. 2019. Controllable list-wise ranking for universal no-reference image quality assessment[EB/OL]. [2021-11-26] . https://arxiv.org/pdf/1911.10566.pdf https://arxiv.org/pdf/1911.10566.pdf

Sellahewa H and Jassim S A. 2010. Image-quality-based adaptive face recognition. IEEE Transactions on Instrumentation and Measurement, 59(4): 805-813[DOI: 10.1109/TIM.2009.2037989]

Sun H H, Shao Z H, Shang Y Y, Chen B and Zhao X X. 2020. Cancelable face recognition with fusion of structural features. Journal of Image and Graphics, 25(12): 2553-2562

孙浩浩, 邵珠宏, 尚媛园, 陈滨, 赵晓旭. 2020. 结构特征下的可撤销人脸识别. 中国图象图形学报, 25(12): 2553-2562)[DOI: 10.11834/jig.190439]

Sun Y, Wang X G and Tang X O. 2014. Deep learning face representation from predicting 10 000 classes//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 1891-1898[DOI: 10.1109/CVPR.2014.244 http://dx.doi.org/10.1109/CVPR.2014.244 ]

Szegedy C, Ioffe S, Vanhoucke V and Alemi A A. 2017. Inception-v4, inception-ResNet and the impact of residual connections on learning//Proceedings of the 31st AAAI Conference on Artificial Intelligence. San Francisco, USA: AAAI Press: 4278-4284

Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V and Rabinovich A. 2015. Going deeper with convolutions//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 1-9[DOI: 10.1109/CVPR.2015.7298594 http://dx.doi.org/10.1109/CVPR.2015.7298594 ]

Taigman Y, Yang M, Ranzato M A and Wolf L. 2014. DeepFace: closing the gap to human-level performance in face verification//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 1701-1708[DOI: 10.1109/CVPR.2014.220 http://dx.doi.org/10.1109/CVPR.2014.220 ]

Terhörst P, Kolf J N, Damer N, Kirchbuchner F and Kuijper A. 2020. SER-FIQ: unsupervised estimation of face image quality based on stochastic embedding robustness//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 5650-5659[DOI: 10.1109/CVPR42600.2020.00569 http://dx.doi.org/10.1109/CVPR42600.2020.00569 ]

Venkatanath N, Praneeth D, Bh M C, Channappayya S S and Medasani S S. 2015. Blind image quality evaluation using perception based features//Proceedings of the 21st National Conference on Communications. Mumbai, India: IEEE: 1-6[DOI: 10.1109/NCC.2015.7084843 http://dx.doi.org/10.1109/NCC.2015.7084843 ]

Wang Z and Bovik A C. 2002. A universal image quality index. IEEE Signal Processing Letters, 9(3): 81-84[DOI: 10.1109/97.995823]

Yi D, Lei Z, Liao S C and Li S Z. 2014. Learning face representation from scratch[EB/OL]. [2021-11-26] . https://arxiv.org/pdf/1411.7923.pdf https://arxiv.org/pdf/1411.7923.pdf

Zhang J W, Shen W and Wu Z D. 2019. Recognition of face privacy protection using convolutional neural networks. Journal of Image and Graphics, 24(5): 744-752

章坚武, 沈炜, 吴震东. 2019. 卷积神经网络的人脸隐私保护识别. 中国图象图形学报, 24(5): 744-752)[DOI: 10.11834/jig.180444]

Zhang L J, Shao X H, Yang F, Deng P L, Zhou X D and Shi Y. 2019. Multi-branch face quality assessment for face recognition//Proceedings of the 19th IEEE International Conference on Communication Technology. Xi'an, China: IEEE: 1659-1664[DOI: 10.1109/ICCT46805.2019.8947255 http://dx.doi.org/10.1109/ICCT46805.2019.8947255 ]

Zhang L J, Zhang L and Li L D. 2017. Illumination quality assessment for face images: a benchmark and a convolutional neural networks based model//Proceedings of the 24th International Conference on Neural Information Processing. Guangzhou, China: Springer: 583-593[DOI: 10.1007/978-3-319-70090-8_59 http://dx.doi.org/10.1007/978-3-319-70090-8_59 ]