发布时间: 2018-10-16 摘要点击次数: 全文下载次数: DOI: 10.11834/jig.180004 2018 | Volume 23 | Number 10 图像理解和计算机视觉

1. 首都师范大学信息工程学院, 北京 100048;
2. 首都师范大学成像技术北京市高精尖创新中心, 北京 100048
 收稿日期: 2018-01-05; 修回日期: 2018-04-08 基金项目: 国家自然科学基金项目（61601311，61603022）；北京市自然科学基金项目（4162017）；北京市优秀人才基金项目（2016000020124G088）；北京市教委科研计划基金项目（SQKM201810028018）；北京市属高校高水平教师队伍建设支持计划基金项目（CIT&TCD20170322） 第一作者简介: 瞿道庆, 1991年生, 男, 硕士研究生, 主要研究方向为生物特征识别。E-mail:dimeraqing@163.com;马琳, 女, 硕士研究生, 主要研究方向为数字图像处理。E-mail:marlene_bj@163.com. 中图法分类号: T 文献标识码: A 文章编号: 1006-8961(2018)10-1540-09

# 关键词

Kinship verification combined with deep metric learning
Qu Daoqing1, Ma Lin1, Shao Zhuhong1,2
1. College of Information Engineering, Capital Normal University, Beijing 100048, China;
2. Beijing Advanced Innovation Center for Imaging Technology, Capital Normal University, Beijing 100048, China
Supported by: National Natural Science Foundation of China(61601311, 61603022)

# Abstract

Objective Facial image analysis has been an important and active subfield of computer vision and pattern recognition.Kinship verification refers to the tasks of training a machine based on features extracted from facial images to determine the existence, type, and degree of kinship of two or more people.Moreover, kinship verification has extensive and promising applications in many technical and social fields, such as family history research and photo management in social network services.In addition, the study on kinship verification is of great theoretical significance to related fields.Several commonalities can be observed among kinship verification algorithms, which are based on traditional machine learning.This paper proposes a new method for studying kinship verification based on face images through deep metric learning. Method Currently, deep learning algorithm can effectively understand a single face image, but the relationship between multiple subjects is still one of the challenging problems in the field of computer vision.This paper presents an approach to kinship verification between parents and children.First, a deep convolution neural network called "FaceCNN" is trained by using a dataset that contains more than 5 000 000 face images from which deep features are extracted.The training set of the deep convolutional neural network contains Internet-sourced datasets of thousands of people, including public kinship datasets, Microsoft face datasets, some films and television dramas, and public figures.Each dataset contains several face images.These face images have complex backgrounds, large differences in image quality, and various face postures, expressions, ages, and genders.Before training the deep network, the face images in the dataset must be preprocessed, including face filtering, face detection, and key point alignment.Then, all images are cut into 144×144 pixels.Then, we divide the father-mother-son and father-mother-daughter samples into father-son, mother-son, father-daughter, and mother-daughter samples.However, we cannot use the Mahalanobis distance to perform metric learning for kinship verification because the traditional Mahalanobis distance measurement learning method only seeks a linear transformation and cannot easily capture the non-linear manifold of the face image.A discriminative metric learning algorithm called DDML is adopted to render the kinship features as near as possible and vice versa.Hierarchical non-linear transformation is employed to achieve a more discriminative learning.Finally, we propose a kinship verification method for parents and children where the cosine similarity is calculated between parents and children.The children may be more similar to one of the parents, and thus we must calculate the similarity probability of the children with their father and mother.The final kinship score is the weighted average of the similarity between children and parents. Result A nine-layer deep model is trained to determine whether the given images have a relationship.The effectiveness of features extraction by using FaceCNN combined with the deep metric learning for kinship verification is validated in the experiment in which the recognition accuracy between parents and son performed on the TSKinFace dataset is 87.71%, and that between parents and daughter is 89.18%, which indicates that the recognition rate of the model increased by 1.31% and 4.87%, respectively.Meanwhile, the proposed approach only requires 3.616 s for metric learning and parental similarity calculation, reducing the time consumed greatly compared with other algorithms. Conclusion The proposed kinship verification takes full advantage of the characterization and learning ability of the deep neural network, and it does not only consume less time but also improves the recognition accuracy.

# Key words

kinship verification; convolution neural network; deep feature; deep learning; metric learning

# 2.1 深度提取网络FaceCNN

Table 1 Parameter settings of FaceCNN

 FaceCNN Configuration input(144×144 RGB image) 5×5 conv，64 2×2，max-pooling & mean-pooling 3×3 conv，64，128 2×2，max-pooling & mean-pooling 3×3 conv，196，256，320 2×2，max-pooling & mean-pooling 3×3 conv，512，512 2×2，max-pooling & mean-pooling 5×5 conv，640 Dropout fc-4 096，1 000 Center Loss & Softmax

# 2.1.1 Maxout层

FaceCNN每一个卷积层后都紧跟一个Maxout层。Maxout本质是一种前向传播网络，它是一个将线性输入转化成非线性输出的小网络结构，可以把它看成是深度卷积网络的激活函数层。一般的激活函数，比如Sigmoid，其隐含层输出表达式

 ${h_i}\left( x \right) = {\rm{sigmoid}}\left( {{\mathit{\boldsymbol{x}}^{\rm{T}}}{\mathit{\boldsymbol{W}}_{ \cdots i}} + {\mathit{\boldsymbol{b}}_i}} \right)$ (1)

 ${h_i}\left( x \right) = \mathop {\max }\limits_{j \in \left[ {1,k} \right]} {\mathit{\boldsymbol{x}}^{\rm{T}}}{\mathit{\boldsymbol{W}}_{ \cdots ij}} + {b_{ij}},{\mathit{\boldsymbol{W}}_{ \cdots ij}} \in {{\bf{R}}^{d \times m \times k}}$ (2)

# 2.1.3 Dropout防止过拟合

FaceCNN在Softmax层前进行Dropout操作，训练深度网络时随机将一定比例的神经元舍弃，而测试时这些神经元仍保留，利用这种方式防止模型过拟合，提高模型泛化能力。

# 2.1.4 CenterLoss代价函数

 $\begin{array}{*{20}{c}} {L = {L_s} + \lambda {L_c} = }\\ { - \sum\limits_{i = 1}^m {\log \frac{{{{\rm{e}}^{W_{{y_i}}^{\rm{T}}{x_i} + {b_{{y_i}}}}}}}{{\sum\limits_{j = 1}^n {{{\rm{e}}^{W_J^{\rm{T}}{x_i} + {b_j}}}} }}} + \frac{\lambda }{2}\sum\limits_{i = 1}^m {\left\| {{x_i} - {c_{{y_i}}}} \right\|_2^2} } \end{array}$ (3)

# 2.3 血缘度量学习

 $d_{{\rm{fc}}}^2\left( {{x_{\rm{f}}},{x_{\rm{c}}}} \right) = \left\| {F\left( {{x_{\rm{f}}}} \right) - F\left( {{x_{\rm{c}}}} \right)_2^2} \right\|$ (5)

 $\left\{ \begin{array}{l} d_{{\rm{fc}}}^2\left( {{x_{\rm{f}}},{x_{\rm{c}}}} \right) < \tau - 1,{x_{\rm{f}}}\;和\;{x_{\rm{c}}}\;有血缘关系\left( {{l_{{\rm{fc}}}} = 1} \right)\\ d_{{\rm{mc}}}^2\left( {{x_{\rm{m}}},{x_{\rm{c}}}} \right) < \tau - 1,{x_{\rm{m}}}\;和\;{x_{\rm{c}}}\;有血缘关系\left( {{l_{{\rm{mc}}}} = 1} \right)\\ d_{{\rm{fc}}}^2\left( {{x_{\rm{f}}},{x_{\rm{c}}}} \right) > \tau + 1,{x_{\rm{f}}}\;和\;{x_{\rm{c}}}\;无血缘关系\left( {{l_{{\rm{fc}}}} = - 1} \right)\\ d_{{\rm{mc}}}^2\left( {{x_{\rm{m}}},{x_{\rm{c}}}} \right) > \tau + 1,{x_{\rm{m}}}\;和\;{x_{\rm{c}}}\;无血缘关系\left( {{l_{{\rm{mc}}}} = - 1} \right) \end{array} \right.$ (6)

 ${l_{{\rm{fc}}}}\left( {\tau - d_{{\rm{fc}}}^2\left( {{x_{\rm{f}}},{x_{\rm{c}}}} \right)} \right) > 1$ (7)

 ${l_{{\rm{mc}}}}\left( {\tau - d_{{\rm{mc}}}^2\left( {{x_{\rm{m}}},{x_{\rm{c}}}} \right)} \right) > 1$ (8)

 $\begin{array}{*{20}{c}} {\arg \mathop {\max }\limits_f J = {J_1} + {J_2} = }\\ {\frac{1}{2}\sum\limits_{{\rm{f}},{\rm{c}}} {g\left( {1 - {l_{{\rm{fc}}}}\left( {\tau - d_{{\rm{fc}}}^2\left( {{x_{\rm{f}}},{x_{\rm{c}}}} \right)} \right)} \right)} + }\\ {\frac{1}{2}\sum\limits_{{\rm{m}},{\rm{c}}} {g\left( {1 - {l_{{\rm{mc}}}}\left( {\tau - d_{{\rm{mc}}}^2\left( {{x_{\rm{m}}},{x_{\rm{c}}}} \right)} \right)} \right)} + }\\ {\frac{\lambda }{2}\sum\limits_{m = 1}^M {\left( {{W^{\left( m \right)}}_F^2 + {b^{\left( m \right)}}_2^2} \right)} } \end{array}$ (9)

 $g\left( z \right) = \frac{1}{\beta }\log \left( {1 + \exp \left( {\beta z} \right)} \right)$ (10)

# 2.4 双亲血缘相似度计算

 ${S_{{\rm{fc}}}}\left( {{{x'}_{\rm{f}}},{{x'}_{\rm{c}}}} \right) = \frac{{\sum\limits_i^d {\left( {{{x'}_{fi}} \times {{x'}_{{\rm{c}}i}}} \right)} }}{{\sqrt {\sum\limits_i^d {{{\left( {{{x'}_{fi}}} \right)}^2}} } \times \sqrt {\sum\limits_i^d {{{\left( {{{x'}_{{\rm{c}}i}}} \right)}^2}} } }}$ (11)

 ${S_{{\rm{mc}}}}\left( {{{x'}_{\rm{m}}},{{x'}_{\rm{c}}}} \right) = \frac{{\sum\limits_i^d {\left( {{{x'}_{{\rm{m}}i}} \times {{x'}_{{\rm{c}}i}}} \right)} }}{{\sqrt {\sum\limits_i^d {{{\left( {{{x'}_{{\rm{m}}i}}} \right)}^2}} } \times \sqrt {\sum\limits_i^d {{{\left( {{{x'}_{{\rm{c}}i}}} \right)}^2}} } }}$ (12)

 ${P_{{\rm{fc}}}} = \frac{{\exp \left( {{S_{{\rm{f}}c}}\left( {{{x'}_{\rm{f}}},{{x'}_{\rm{c}}}} \right)} \right)}}{{\exp \left( {{S_{{\rm{fc}}}}\left( {{{x'}_{\rm{f}}},{{x'}_{\rm{c}}}} \right)} \right) + \exp \left( {{S_{{\rm{mc}}}}\left( {{{x'}_{\rm{m}}},{{x'}_{\rm{c}}}} \right)} \right)}}$ (13)

 ${P_{{\rm{mc}}}} = \frac{{\exp \left( {{S_{{\rm{mc}}}}\left( {{{x'}_{\rm{m}}},{{x'}_{\rm{c}}}} \right)} \right)}}{{\exp \left( {{S_{{\rm{fc}}}}\left( {{{x'}_{\rm{f}}},{{x'}_{\rm{c}}}} \right)} \right) + \exp \left( {{S_{{\rm{mc}}}}\left( {{{x'}_{\rm{m}}},{{x'}_{\rm{c}}}} \right)} \right)}}$ (14)

 $S = {P_{{\rm{fc}}}}{S_{{\rm{fc}}}} + {P_{{\rm{mc}}}}{S_{{\rm{mc}}}}$ (15)

# 3.3 实验结果

DCNML算法在家庭血缘数据集TSKinFace上进行实验获得的ROC曲线如图 7所示。DCNML在双亲血缘关系识别问题上取得了较为理想的识别准确率(如表 2)，儿子与父母的识别准确率可以达到87.71%，女儿与父母的识别准确率可以达到89.18%。同时，在均不考虑特征提取阶段耗时的情况下，DCNML耗时比现有的RSBM[18]方法少很多，如表 3所示。

Table 2 Comparison of recognition accuracy

 /% 算法 父母—儿子 父母—女儿 NRML[12] 77.0 71.4 DDML[17] 82.1 79.8 ABM[18] 78.5 73.2 ABM-block[18] 83.4 81.9 SBM[18] 82.4 78.2 SBM-block[18] 85.2 83.5 RSBM-block[18] 86.4 84.4 DCNN[13] 76.6 71.4 HRMFS[19] 76.9 77.3 EHRMFS[19] 80.2 80.16 Gate Autoencoder[20] 81.9 79.6 本文 87.71 89.18 注：黑体为最优结果。

Table 3 Comparison of computational efficiency

 /s 算法 计算耗时 测试耗时 本文 3.616 0.314 RSBM[18] 17 385.610 0.404

# 参考文献

• [1] Fang R G, Tang K D, Snavely N, et al.Towards computational models of kinship verification[C]//Proceedings of 2010 IEEE International Conference on Image Processing.Hong Kong, China: IEEE, 2010: 1577-1580.[DOI:DOI:10.1109/ICIP.2010.5652590]
• [2] Xia S Y, Shao M, Luo J B, et al. Understanding kin relationships in a photo[J]. IEEE Transactions on Multimedia, 2012, 14(4): 1046–1056. [DOI:10.1109/TMM.2012.2187436]
• [3] Qin X Q, Liu D K, Wang D. Heterogeneous similarity learning for more practical kinship verification[J]. Neural Processing Letters, 2018, 47(3): 1253–1269. [DOI:10.1007/s11063-017-9694-3]
• [4] Zhao Y G, Song Z, Zheng F, et al. Learning a multiple kernel similarity metric for kinship verification[J]. Information Sciences, 2018, 430-431: 247–260. [DOI:10.1016/j.ins.2017.11.048]
• [5] Sun Y, Chen Y H, Wang X G, et al.Deep learning face representation by joint identification-verification[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems.Montreal, Canada: MIT Press, 2014: 1988-1996.
• [6] Somanath G, Kambhamettu C.Can faces verify blood-relations?[C]//Proceedings of the IEEE 5th International Conference on Biometrics: Theory, Applications and Systems.Arlington, VA, USA: IEEE Press, 2012: 105-112.[DOI:10.1109/BTAS.2012.6374564]
• [7] Ghahramani M, Yau W Y, Teoh E K. Family verification based on similarity of individual family member's facial segments[J]. Machine Vision and Applications, 2014, 25(4): 919–930. [DOI:10.1007/s00138-013-0566-1]
• [8] Dibeklioglu H, Salah A A, Gevers T.Like father, like son: facial expression dynamics for kinship verification[C]//Proceedings of 2013 IEEE International Conference on Computer Vision.Sydney, Australia: IEEE, 2013: 1497-1504.[DOI:10.1109/ICCV.2013.189]
• [9] Vieira T F, Bottino A, Islam I U.Automatic verification of parent-child pairs from face images[C]//Proceedings of the 18th Iberoamerican Congress Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications.Havana, Cuba: Springer-Verlag, 2013: 326-333.[DOI:10.1007/978-3-642-41827-3_41]
• [10] Vieira T F, Bottino A, Laurentini A, et al. Detecting siblings in image pairs[J]. The Visual Computer, 2014, 30(12): 1333–1345. [DOI:10.1007/s00371-013-0884-3]
• [11] Fang R G, Gallagher A C, Chen T, et al.Kinship classification by modeling facial feature heredity[C]//Proceedings of 2013 IEEE International Conference on Image Processing.Melbourne, VIC, Australia: IEEE Press, 2014: 2983-2987.[DOI:10.1109/ICIP.2013.6738614]
• [12] Lu J W, Zhou X Z, Tan Y P, et al. Neighborhood repulsed metric learning for kinship verification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(2): 331–345. [DOI:10.1109/TPAMI.2013.134]
• [13] Zhang K P, Zhang Z P, Li Z F, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499–1503. [DOI:10.1109/LSP.2016.2603342]
• [14] Goodfellow I, Warde-Farley D, Mirza M, et al. Maxout networks[C]//Proceddings of the 30th Ineternational Conference on Machine Learning. PMLR, 2013: 1319-1327..
• [15] Wen Y D, Zhang K P, Li Z F, et al.A discriminative feature learning approach for deep face recognition[C]//Proceedings of the 14th European Conference on Computer Vision.Amsterdam, The Netherlands: Springer International Publishing, 2016: 499-515.[DOI:10.1007/978-3-319-46478-7_31]
• [16] van der Maaten L, Hinton G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579–2605.
• [17] Hu J L, Lu J W, Tan Y P.Discriminative deep metric learning for face verification in the wild[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition.Columbus, OH, USA: IEEE, 2014: 1875-1882.[DOI:10.1109/CVPR.2014.242]
• [18] Qin X Q, Tan X Y, Chen S C. Tri-subject kinship verification:understanding the core of a family[J]. IEEE Transactions on Multimedia, 2015, 17(10): 1855–1867. [DOI:10.1109/TMM.2015.2461462]
• [19] Dehshibi M M, Shanbehzadeh J. Cubic norm and kernel-based bi-directional PCA:toward age-aware facial kinship verification[J]. The Visual Computer, 2017: 1–18. [DOI:10.1007/s00371-017-1442-1]
• [20] Dehghan A, Ortiz E G, Villegas R, et al.Who do i look like? determining parent-offspring resemblance via gated autoencoders[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition.Columbus, OH, USA: IEEE, 2014: 1757-1764.[DOI:10.1109/CVPR.2014.227]
• [21] Kohli N, Vatsa M, Singh R, et al. Hierarchical representation learning for kinship verification[J]. IEEE Transactions on Image Processing, 2017, 26(1): 289–302. [DOI:10.1109/TIP.2016.2609811]