浏览全部资源
扫码关注微信
纸质出版日期： 2021-02-16 ，
录用日期： 2020-05-18
移动端阅览
张茜, 刘骊, 甘霖, 付晓东, 刘利军, 黄青松. 融合视觉风格和标签约束的少数民族服装图像解析[J]. 中国图象图形学报, 2021,26(2):402-414.
Qian Zhang, Li Liu, Lin Gan, Xiaodong Fu, Lijun Liu, Qingsong Huang. Clothing parsing of Chinese minorities via the fusion of visual style and label constraints[J]. Journal of Image and Graphics, 2021,26(2):402-414.
目的
2
少数民族服装款式结构复杂，视觉风格各异。由于缺少民族服装语义标签、局部特征繁杂以及语义标签之间存在相互干扰等因素导致少数民族服装图像解析准确率和精度较低。因此，本文提出了一种融合视觉风格和标签约束的少数民族服装图像解析方法。
方法
2
首先基于本文构建的包含55个少数民族的服装图像数据集，按照基本款式结构、着装区域、配饰和不同视觉风格自定义少数民族服装的通用语义标签和民族语义标签，同时设置4组标注对，共8个标注点；然后，结合自定义语义标签和带有标注对的训练图像，在深度完全卷积神经网络SegNet中加入视觉风格以融合局部特征和全局特征，并引入属性预测、风格预测和三元组损失函数对输入的待解析图像进行初步解析；最后，通过构建的标签约束网络进一步优化初步解析结果，避免标签相互干扰，得到优化后的最终解析结果。
结果
2
在构建的少数民族服装图像数据集上进行验证，实验结果表明，标注对有效提升了局部特征的检测准确率，构建的视觉风格网络能够有效融合少数民族服装的全局特征和局部特征，标签约束网络解决了标签之间相互干扰的问题，在结合视觉风格网络和标签约束网络后，能够明显提升少数民族服装解析的平均精度，像素准确度达到了90.54%。
结论
2
本文提出的融合视觉风格和标签约束的少数民族服装图像解析方法，能够提高少数民族服装图像解析的准确率和精度，对传承祖国文化、保护非物质文化遗产具有很好的意义。
Objective
2
Many minority groups live in China
and the visual styles of their clothing are different. The combination of clothing parsing and the clothing culture of these minority groups plays an important role in realizing the digital protection of the clothing images of these groups and the inheritance of their culture. However
a complete dataset of the clothing images of Chinese minorities remains lacking. The clothing styles of minority groups have complex structures and different visual styles. Semantic labels to distinguish the clothing of different minorities are lacking
and defining the semantic labels of ethnic accessories is a challenging task. Describing information
such as local details
styles
and ethnic characteristics of minority group clothing
is difficult when using existing clothing image parsing methods. Mutual interference between semantic labels leads to unsatisfactory accuracy and precision of clothing image parsing. Therefore
we proposed a clothing parsing method based on visual style and label constraints.
Method
2
Our method primarily parsed minority group clothing through their visual style by fusing local and global features. Then
the label constraint network was used to suppress redundant tags and optimized the preliminary parsing results. First
we defined the general semantic labels of minority group clothing. The distinctive semantic labels were defined in accordance with the combination preference of semantic labels. We set four sets of annotation pairs based on human body parts
with a total of eight label points. Each pair of annotations corresponds to a set of key points on the clothing structure. The upper body garment was marked with the left/right collar
left/right sleeves
and left/right top hem. The lower body garment was marked with the left/right bottom hem. We also marked the visibility of each annotation and used the label annotations to determine whether occlusion occurred in the clothing. Second
combining the training images with the annotation pairs and the self-defined semantic labels
a visual style network was added on the basis of a full convolutional network. A branch was built on the last convolutional layer in the SegNet network. The branch was divided into three parts
with each part respectively dealing with the position and visibility of the annotation pairs and the local and global characteristics of the clothes. The two parts of the local and global features of the clothing were outputted to "fc7_fusion" for fusion. The style features were returned to the SegNet network through a deconvolution layer
and preliminary parsing results were obtained. Finally
a label mapping function was used to convert the preliminary parsing result into a label vector in accordance with the number of labels. Each element indicates whether a corresponding label exists in the preliminary parsing result. Then
the label vector was compared with the true semantic labels in the training set
and the labels were corrected to suppress redundant label probability scores. The label constraint network eliminated redundant and erroneous labels by comparing the labels of the preliminary parsing results with those of the training images. The label constraint network avoided the mutual interference of labels and increased the accuracy of the parsing result. In addition
we constructed a clothing image dataset of 55 minority groups. The primary sources were online shopping sites
such as Taobao
Tmall
and JD. This dataset was expanded by including datasets from other platforms
such as Baidu Pictures
blogs
and forums. A total of 61 710 images were collected. At least 500 images were collected for each minority group.
Result
2
The proposed method was validated on an image dataset of minority group clothing. Experimental results showed that the detection accuracy of clothing visual style features was higher with annotation pairs. The visual style network efficiently fused local and global features. The label constraint network effectively solved the mutual interference problem of labels. The method proposed in this study improved parsing accuracy on large-scale clothing labels
particularly on skirts with considerable differences in pattern texture and color blocks. The method also improved the small labels of accessories
such as hats and collars. The results of the minority group clothing parsing improved significantly. The pixel accuracy of the parsing results reached 90.54%.
Conclusion
2
The clothing of minority groups is characterized by complicated styles and accessories
lack of semantic labels
and complex labels that interfere with one another. Thus
we proposed a clothing parsing method that fuses visual style with label constraints. We constructed a dataset of minority group clothing images and defined the generic and distinctive semantic labels of minority group clothing. We made pixel-level semantic annotations and set up annotation pairs on the training images. Then
we built a visual style network based on SegNet to obtain preliminary parsing results. Finally
the mutual interference problem of semantic labels was solved through a label constraint network to obtain the final parsing result. Compared with other clothing parsing methods
our method improved the accuracy of minority group clothing image parsing. Inheriting culture and protecting intangible cultural heritage are significant. However
some clothing parsing results of this method are not ideal
particularly the accuracy of small accessories. The semantic labels of minority group clothing are imperfect and insufficiently accurate. Subsequent work will continue to improve the dataset
focusing on the aforementioned issues to further improve the accuracy of minority group clothing parsing.
少数民族服装图像解析语义标签视觉风格标签约束
minority clothingimage parsingsemantic labelsvisual stylelabel constraints
Badrinarayanan V, Kendall A and Cipolla R. 2017. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12): 2481-2495[DOI: 10.1109/TPAMI.2016.2644615]
Guo J M, Su Z, Luo X H, Zhang G W and Liang X W. 2018. Conditional feature coupling network for multi-persons clothing parsing//Proceedings of the 19th Pacific-Rim Conference on Multimedia. Hefei, China: Springer: 189-200[DOI:10.1007/978-3-030-00776-8_18http://dx.doi.org/10.1007/978-3-030-00776-8_18]
Hidayati S C, You C W, Cheng W H and Hua K L. 2018. Learning and recognition of clothing genres from full-body images. IEEE Transactions on Cybernetics, 48(5): 1647-1659[DOI: 10.1109/TCYB.2017.2712634]
Huo P, Wang Y H and Liu Q J. 2016. A part-based and feature fusion method for clothing classification//Proceedings of the 17th Pacific-Rim Conference on Multimedia. Xi'an, China: Springer: 231-241[DOI:10.1007/978-3-319-48890-5_23http://dx.doi.org/10.1007/978-3-319-48890-5_23]
Jammalamadaka N, Minocha A, Singh D and Jawahar C V. 2013. Parsing clothes in unrestricted images//Proceedings of British Machine Vision Conference. Bristol, UK: BMVA Press: #88[DOI:10.5244/C.27.88http://dx.doi.org/10.5244/C.27.88]
Ji J and Yang R Y. 2017. An improved clothing parsing method emphasizing the clothing with complex texture//Proceedings of the 18th Pacific-Rim Conference on Multimedia. Harbin, China: Springer: 487-496[DOI:10.1007/978-3-319-77380-3_46http://dx.doi.org/10.1007/978-3-319-77380-3_46]
Ji W, Li X, Zhuang Y T, El Farouk Bourahla O, Ji Y X, Li S H and Cui J B. 2018. Semantic locality-aware deformable network for clothing segmentation//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm, Sweden: IJCAI: 764-770[DOI:10.24963/ijcai.2018/106http://dx.doi.org/10.24963/ijcai.2018/106]
Khurana T, Mahajan K, Arora C and Rai A. 2018. Exploiting texture cues for clothing parsing in fashion images//Proceedings of the 25th IEEE International Conference on Image Processing. Athens, Greece: IEEE: 2102-2106[DOI:10.1109/ICIP.2018.8451281http://dx.doi.org/10.1109/ICIP.2018.8451281]
Kiapour M H, Yamaguchi K, Berg A C and Berg T L. 2014. Hipster wars: discovering elements of fashion styles//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 472-488[DOI:10.1007/978-3-319-10590-1_31http://dx.doi.org/10.1007/978-3-319-10590-1_31]
Liang X D, Lin L, Yang W, Luo P, Huang J S and Yan S C. 2016a. Clothes co-parsing via joint image segmentation and labeling with application to clothing retrieval. IEEE Transactions on Multimedia, 18(6): 1175-1186[DOI: 10.1109/TMM.2016.2542983]
Liang X D, Liu S, Shen X H, Yang J C, Liu L Q, Dong J, Lin L and Yan S C. 2015a. Deep human parsing with active template regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(12): 2402-2414[DOI: 10.1109/TPAMI.2015.2408360]
Liang X D, Shen X H, Feng J S, Lin L and Yan S C. 2016b. Semantic object parsing with graph LSTM//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 125-143[DOI:10.1007/978-3-319-46448-0_8http://dx.doi.org/10.1007/978-3-319-46448-0_8]
Liang X D, Xu C Y, Shen X H, Yang J C, Tang J H, Lin L and Yan S C. 2015b. Human parsing with contextualized convolutional neural network//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 1386-1394[DOI:10.1109/ICCV.2015.163http://dx.doi.org/10.1109/ICCV.2015.163]
Liu S, Liang X D, Liu L Q, Shen X H, Yang J C, Xu C S, Lin L, Cao X C and Yan S C. 2015. Matching-CNN meets KNN: quasi-parametric human parsing//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 1419-1427[DOI:10.1109/CVPR.2015.7298748http://dx.doi.org/10.1109/CVPR.2015.7298748]
Liu Z W, Luo P, Qiu S, Wang X G and Tang X O. 2016. DeepFashion: powering robust clothes recognition and retrieval with rich annotations//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 1096-1104[DOI:10.1109/CVPR.2016.124http://dx.doi.org/10.1109/CVPR.2016.124]
Russell B C, Torralba A, Murphy K P and Freeman W T. 2008. LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 77(1/3): 157-173[DOI: 10.1007/s11263-007-0090-8]
Simo-Serra E, Fidler S, Moreno-Noguer F and Urtasun R. 2014. A high performance CRF model for clothes parsing//Proceedings of the 12th Asian Conference on Computer Vision. Singapore, Singapore: Springer: 64-81[DOI:10.1007/978-3-319-16811-1_5http://dx.doi.org/10.1007/978-3-319-16811-1_5]
Simo-Serra E and Ishikawa H. 2016. Fashion style in 128 floats: joint ranking and classification using weak data for feature extraction//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 298-307[DOI:10.1109/CVPR.2016.39http://dx.doi.org/10.1109/CVPR.2016.39]
Su Z, Guo J M, Zhang G W, Luo X H, Wang R M and Zhou F. 2019. Conditional progressive network for clothing parsing. IET Image Processing, 13(4): 556-565[DOI: 10.1049/iet-ipr.2018.5494]
Vaccaro K, Shivakumar S, Ding Z Q, Karahalios K and Kumar R. 2016. The elements of fashion style//Proceedings of the 29th Annual Symposium on User Interface Software and Technology. Tokyo, Japan, ACM: 777-785[DOI:10.1145/2984511.2984573http://dx.doi.org/10.1145/2984511.2984573]
Vittayakorn S, Umeda T, Murasaki K, Sudo K, Okatani T and Yamaguchi K. 2016. Automatic attribute discovery with neural activations//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 252-268[DOI:10.1007/978-3-319-46493-0_16http://dx.doi.org/10.1007/978-3-319-46493-0_16]
Wu S M, Liu L, Fu X D, Liu L J and Huang Q S. 2019. Human detection and multi-task learning for minority clothing recognition. Journal of Image and Graphics, 24(4): 562-572
吴圣美, 刘骊, 付晓东, 刘利军, 黄青松. 2019.结合人体检测和多任务学习的少数民族服装识别.中国图象图形学报, 24(4): 562-572)[DOI: 10.11834/jig.180500]
Yamaguchi K, Kiapour M H and Berg T L. 2014. Paper doll parsing: retrieving similar styles to parse clothing items//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 3519-3526[DOI:10.1109/ICCV.2013.437http://dx.doi.org/10.1109/ICCV.2013.437]
Yamaguchi K, Kiapour M H, Ortiz L E and Berg T L. 2012. Parsing clothing in fashion photographs//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA: IEEE: 3570-3577[DOI:10.1109/CVPR.2012.6248101http://dx.doi.org/10.1109/CVPR.2012.6248101]
Yamaguchi K, Okatani T, Sudo K, Murasaki K and Taniguchi Y. 2015. Mix and match: joint model for clothing and attribute recognition//Proceedings of British Machine Vision Conference (BMVC). Swansea, UK: BMVA Press: 51.1-51.12[DOI:10.5244/C.29.51http://dx.doi.org/10.5244/C.29.51]
Zheng Q, Chen J, Liang C, Fang W H, Jing X Y and Hu R M. 2017. Transferring clothing parsing from fashion dataset to surveillance//Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing. New Orleans, USA: IEEE: 1667-1671[DOI:10.1109/ICASSP.2017.7952440http://dx.doi.org/10.1109/ICASSP.2017.7952440]
Zhong M L and Fan P. 2006. Chinese Minority Costumes. Beijing: China Textile Press
钟茂兰, 范朴. 2006.中国少数民族服饰.北京:中国纺织出版社
京公网安备11010802024621