浏览全部资源
扫码关注微信
纸质出版日期： 2023-04-16 ，
移动端阅览
汪芸竹， 刘骊， 付晓东， 刘利军， 黄青松. 2023. 全局—局部特征优化的时尚服饰搭配. 中国图象图形学报， 28(04):1104-1118
Wang Yunzhu， Liu Li， Fu Xiaodong， Liu Lijun， Huang Qingsong. 2023. Fashion clothing matching by global-local feature optimization. Journal of Image and Graphics， 28(04):1104-1118
目的
2
由于现有时尚服饰搭配方法缺乏服饰图像局部细节的有效特征表示，难以对不同服饰间的局部兼容性进行建模，限制了服饰兼容性学习的完备性，导致时尚服饰搭配的准确率较低。因此，提出一种全局—局部特征优化的时尚服饰搭配方法。
方法
2
首先，利用不同卷积网络提取时尚服饰的图像和文本特征作为全局特征，同时在卷积网络基础上构建局部特征提取网络，提取时尚服饰图像的局部特征；然后，基于图网络和自注意力机制构建全局—局部兼容性学习模块，通过学习不同时尚服饰全局特征间和局部特征间的交互关系，并定义不同时尚服饰的权重，进行服饰全局和局部兼容性建模；最后，构建服饰搭配优化模型，通过融合套装中所有服饰的全局和局部兼容性优化服饰搭配，并计算搭配得分，输出正确的服饰搭配结果。
结果
2
在公开数据集Polyvore上将本文方法与其他方法进行对比。实验结果表明，利用局部特征提取网络提取的时尚服饰图像局部特征能有效地表示服饰局部信息；构建的全局—局部兼容性学习模块对时尚服饰的全局兼容性和局部兼容性进行了完整建模；构建的时尚服饰搭配优化模型实现了全局和局部兼容性的优化组合，使时尚服饰搭配准确率（fill in the blank，FITB）提高至86.89%。
结论
2
本文提出的全局—局部特征优化的时尚服饰搭配方法，能够有效提高时尚服饰搭配的准确率，较好地满足日常时尚搭配的需求。
Objective
2
Fashion clothing matching has been developing for clothing-relevant fashion research nowadays. Fashion clothing matching studies are required to learn the complex matching relationship （i.e.， fashion compatibility） among different fashion items in an representation-based outfit. Fashion items have rich partial designs and matching relationships among partial designs. To analyze their global compatibility learning， most of the existing researches are concerned of items’ global features （visual and textual features）. But， local feature extraction is often ignored for local compatibility， which causes lower performance and accuracy of fashion style matching. Therefore， we develop a fashion style matching method in terms of global-local feature optimization and it is aimed to extract the local features of fashion images for local information representing， construct the local compatibility of fashion items， and improve the global and local compatibility-incorporated accuracy of fashion style matching.
Method
2
First， we use two different convolutional neural networks （CNNs） to extract the global features of fashion items separately on the basis of the input fashion images and texts. To extract CNN-based local features of fashion images， a multiple branches-related local feature extraction network is designed. A branch of the local feature extraction network is composed of 1） a convolution layer， 2） a batch normalization （BN） layer， and 3） a rectified linear unit （ReLU） activation function. A branch can be used to extract a local feature in the fashion image， and different branches can be used to extract different local features of the fashion image. Second， a global-local compatibility learning module is constructed in terms of graph neural network （GNN） and self-attention mechanism （SAM）， which can model both of the global and local compatibility. GNN is used to model interactions among global features and local features separately. The SAM-based weight information of different fashion items is defined and integrated into the modeling， and the item’s global and local compatibility are obtained both. Finally， a fashion clothing matching optimization model is built up to gain optimized matching results. The learned outfit global and local compatibility can be used to integrate all fashion items’ global compatibility and local compatibility separately in an outfit. To optimize matching results， the trade-off parameters are then defined to adjust the impact of the outfit global compatibility and local compatibility on fashion style matching. At the same time， the matching score is calculated as well. Different matching schemes have different matching scores， and the optimized fashion style matching result is generated according to the highest score.
Result
2
The proposed method is validated on the public Polyvore dataset that includes fashion item images and textual descriptions. The details are presented as follows. The local features of fashion items extracted by our local feature extraction network can represent the fashion items’ local information effectively without label-attributed supervision. Our global-local compatibility learning module can be used to learn the fashion item’s global compatibility and local compatibility at the same time， and the weights of different fashion items is involved， which can model the fashion global and local compatibility completely. The fill in the blank （FITB） accuracy ratio of fashion style matching is improved to 86.89%.
Conclusion
2
A fashion clothing matching method is developed in terms of global local feature optimization. First， we construct a local feature extraction network to extract local features of fashion images while the global features of fashion items are extracted. Next， the self-attention mechanism is introduced to weight different fashion items after the global matching relationships and local matching relationships of fashion items with graph network are analyzed， which constructs the global and local compatibilities of fashion items completely. Finally， to obtain the global and local compatibilities of the outfit， our fashion clothing matching optimization model is used to fuse the item’s global compatibility and local compatibility each in an outfit. To optimize matching results， the effectiveness of the two kinds of compatibilities on fashion clothing matching with parameters is adjusted as well. The convergence speed of our method is still slow. The optimization model is only used to combine the global and local compatibility of the outfit linearly. In practice， the relationship between the global compatibility and local compatibility is more complex. To improve the accuracy of fashion clothing matching， future work can be focused on the convergence speed and the clothing matching optimization further.
时尚服饰搭配兼容性建模全局特征局部特征特征优化
fashion clothing matchingcompatibility modelingglobal featurelocal featurefeature optimization
Chen W， Huang P P， Xu J M， Guo X， Guo C， Sun F， Li C， Pfadler A， Zhao H and Zhao B Q. 2019. POG： personalized outfit generation for fashion recommendation at Alibaba iFashion//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Anchorage， USA： ACM： 2662-2670 ［DOI： 10.1145/3292500.3330652http://dx.doi.org/10.1145/3292500.3330652］
Cheng W H， Song S J， Chen C Y， Hidayati S C and Liu J Y. 2022. Fashion meets computer vision： a survey. ACM Computing Surveys， 54（4）： #72 ［DOI： 10.1145/3447239http://dx.doi.org/10.1145/3447239］
Cucurull G， Taslakian P and Vazquez D. 2019. Context-aware visual compatibility prediction//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 12609-12618 ［DOI： 10.1109/CVPR.2019.01290http://dx.doi.org/10.1109/CVPR.2019.01290］
Cui Z Y， Li Z K， Wu S， Zhang X Y and Wang L. 2019. Dressing as a whole： outfit compatibility learning based on node-wise graph neural networks//Proceedings of 2019 World Wide Web Conference. San Francisco， USA： ACM： 307-317 ［DOI： 10.1145/3308558.3313444http://dx.doi.org/10.1145/3308558.3313444］
Dong X， Wu J L， Song X M， Dai H J and Nie L Q. 2020. Fashion compatibility modeling through a multi-modal try-on-guided scheme//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Xi’an， China： ACM： 771-780 ［DOI： 10.1145/3397271.3401047http://dx.doi.org/10.1145/3397271.3401047］
Han X T， Wu Z X， Jiang Y G and Davis L S. 2017. Learning fashion compatibility with bidirectional LSTMs//Proceedings of the 25th ACM International Conference on Multimedia. Mountain View， USA： ACM： 1078-1086 ［DOI： 10.1145/3123266.3123394http://dx.doi.org/10.1145/3123266.3123394］
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Iwata T， Watanabe S and Sawada H. 2011. Fashion coordinates recommender system using photographs from fashion magazines//Proceedings of the 22nd International Joint Conference on Artificial Intelligence. Barcelona， Spain： AAAI： 2262-2267
Li X C， Wang X， He X N， Chen L， Xiao J and Chua T S. 2020a. Hierarchical fashion graph network for personalized outfit recommendation//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Xi’an， China： ACM： 159-168 ［DOI： 10.1145/3397271.3401080http://dx.doi.org/10.1145/3397271.3401080］
Li Y， Luo Y D and Huang Z. 2020b. Graph-based relation-aware representation learning for clothing matching//Proceedings of the 31st Australasian Database Conference on Databases Theory and Applications. Melbourne， Australia： Springer： 189-197 ［DOI： 10.1007/978-3-030-39469-1_15http://dx.doi.org/10.1007/978-3-030-39469-1_15］
Li Z， Wu B， Liu Q， Wu L K， Zhao H K and Mei T. 2020c. Learning the compositional visual coherence for complementary recommendations//Proceedings of the 29th International Joint Conference on Artificial Intelligence. Yokohama， Japan： Morgan Kaufmann： 3536-3543 ［DOI： 10.24963/ijcai.2020/485http://dx.doi.org/10.24963/ijcai.2020/485］
Li Z K， Cui Z Y， Wu S， Zhang X Y and Wang L. 2019. Semi-supervised compatibility learning across categories for clothing matching//Proceedings of 2019 IEEE International Conference on Multimedia and Expo （ICME）. Shanghai， China： IEEE： 484-489 ［DOI： 10.1109/icme.2019.00090http://dx.doi.org/10.1109/icme.2019.00090］
Lin Y L， Tran S and Davis L S. 2020a. Fashion outfit complementary item retrieval//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 3308-3316 ［DOI： 10.1109/CVPR42600.2020.00337http://dx.doi.org/10.1109/CVPR42600.2020.00337］
Lin Y S， Moosaei M and Yang H. 2020b. OutfitNet： fashion outfit recommendation with attention-based multiple instance learning//Proceedings of the Web Conference 2020. Taipei， China： ACM： 77-87 ［DOI： 10.1145/3366423.3380096http://dx.doi.org/10.1145/3366423.3380096］
Liu S， Feng J S， Song Z， Zhang T Z， Lu H Q， Xu C S and Yan S C. 2012. Hi， magic closet， tell me what to wear！//Proceedings of the 20th ACM international conference on Multimedia （MM）. Nara， Japan： ACM： 619-628 ［DOI： 10.1145/2393347.2393433http://dx.doi.org/10.1145/2393347.2393433］
Liu X H， Zhao H Y， Tian M Q， Sheng L， Shao J， Yi S， Yan J J and Wang X G. 2017. HydraPlus-Net： Attentive deep features for pedestrian analysis//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 350-359 ［DOI： 10.1109/iccv.2017.46http://dx.doi.org/10.1109/iccv.2017.46］
Liu Y J， Feng S H， Li Z M and Li H. 2019. FMatchNet algorithm for fast clothing matching. Journal of Image and Graphics， 24（6）： 979-986
刘玉杰， 冯士贺， 李宗民， 李华. 2019. 用于快速服装搭配的FMatchNet算法. 中国图象图形学报， 24（6）： 979-986 ［DOI： 10.11834/jig.180538http://dx.doi.org/10.11834/jig.180538］
Liu Y J， Gao Y B， Bian L Y， Wang W Y and Li Z M. 2018. How to wear beautifully？ Clothing pair recommendation. Journal of Computer Science and Technology， 33（3）： 522-530 ［DOI： 10.1007/s11390-018-1836-1http://dx.doi.org/10.1007/s11390-018-1836-1］
McAuley J， Targett C， Shi Q F and van den Hengel A. 2015. Image-based recommendations on styles and substitutes//Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. Santiago， Chile： ACM： 43-52 ［DOI： 10.1145/2766462.2767755http://dx.doi.org/10.1145/2766462.2767755］
Pang K C， Zou X X and Wai Keung W. 2021. Modeling fashion compatibility with explanation by using bidirectional LSTM//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPRW）. Nashville， USA： IEEE： 3889-3893 ［DOI： 10.1109/cvprw53098.2021.00432http://dx.doi.org/10.1109/cvprw53098.2021.00432］
Shin Y G， Yeo Y J， Sagong M C， Ji S W and Ko S J. 2019. Deep fashion recommendation system with style feature decomposition//Proceedings of the 9th IEEE International Conference on Consumer Electronics （ICCE-Berlin）. Berlin， Germany： IEEE： #8966228 ［DOI： 10.1109/ICCE-Berlin47944.2019.8966228http://dx.doi.org/10.1109/ICCE-Berlin47944.2019.8966228］
Song X M， Feng F L， Liu J H， Li Z K， Nie L Q and Ma J. 2017. NeuroStylist： neural compatibility modeling for clothing matching//Proceedings of the 25th ACM International Conference on Multimedia （MM）. Mountain View， USA： ACM： 753-761 ［DOI： 10.1145/3123266.3123314http://dx.doi.org/10.1145/3123266.3123314］
Song X M， Han X J， Li Y K， Chen J Y， Xu X S and Nie L Q. 2019. GP-BPR： personalized compatibility modeling for clothing matching//Proceedings of the 27th ACM International Conference on Multimedia. Nice， France： ACM： 320-328 ［DOI： 10.1145/3343031.3350956http://dx.doi.org/10.1145/3343031.3350956］
Tan R， Vasileva M I， Saenko K and Plummer B. 2019. Learning similarity conditions without explicit supervision//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 10372-10381 ［DOI： 10.1109/ICCV.2019.01047http://dx.doi.org/10.1109/ICCV.2019.01047］
Teney D， Liu L Q and van Den Hengel A. 2017. Graph-structured representations for visual question answering//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Honolulu， USA： IEEE： 3233-3241 ［DOI： 10.1109/CVPR.2017.344http://dx.doi.org/10.1109/CVPR.2017.344］
Vasileva M I， Plummer B A， Dusad K， Rajpal S， Kumar R and Forsyth D. 2018. Learning type-aware embeddings for fashion compatibility//Proceedings of the 15th European Conference on Computer Vision （ECCV）. Munich， Germany： Springer： 405-421 ［DOI： 10.1007/978-3-030-01270-0_24http://dx.doi.org/10.1007/978-3-030-01270-0_24］
Veit A， Kovacs B， Bell S， McAuley J， Bala K and Belongie S. 2015. Learning visual clothing style with heterogeneous dyadic co-occurrences//Proceedings of 2015 IEEE International Conference on Computer Vision （ICCV）. Santiago， Chile： IEEE： #527 ［DOI： 10.1109/ICCV.2015.527http://dx.doi.org/10.1109/ICCV.2015.527］
Wang R S， Li Z， Cao J， Chen T and Wang L. 2019. Convolutional recurrent neural networks for text classification//Proceedings of 2019 International Joint Conference on Neural Networks （IJCNN）. Budapest， Hungary： IEEE： 1-6 ［DOI： 10.1109/IJCNN.2019.8852406http://dx.doi.org/10.1109/IJCNN.2019.8852406］
Yang X， Song X M， Feng F L， Wen H K， Duan L Y and Nie L Q. 2021. Attribute-wise explainable fashion compatibility modeling. ACM Transactions on Multimedia Computing， Communications， and Applications， 17（1）： #36 ［DOI： 10.1145/3425636http://dx.doi.org/10.1145/3425636］
Yang Z Y， Xue W L， Zhang C X， Ding Y and Ma Y X. 2021. Recommendations for user's bottoms matching based on generative adversarial networks. Journal of Textile Research， 42（7）： 164-168
杨争妍， 薛文良， 张传雄， 丁亦， 马颜雪. 2021. 基于生成式对抗网络的用户下装搭配推荐. 纺织学报， 42（7）： 164-168 ［DOI： 10.13475/j.fzxb.20200803505http://dx.doi.org/10.13475/j.fzxb.20200803505］
Zhang H J， Huang W， Liu L L and Chow T W S. 2020. Learning to match clothing from textual feature-based compatible relationships. IEEE Transactions on Industrial Informatics， 16（11）： 6750-6759 ［DOI： 10.1109/tii.2019.2924725http://dx.doi.org/10.1109/tii.2019.2924725］
Zhou Q Q， Liu L， Liu L J， Fu X D and Huang Q S. 2021. Global-local feature extraction method for fine-grained national clothing image retrieval. Pattern Recognition and Artificial Intelligence， 34（5）： 463-472
周前前， 刘骊， 刘利军， 付晓东， 黄青松. 2021. 细粒度民族服饰图像检索的全局—局部特征提取方法. 模式识别与人工智能， 34（5）： 463-472 ［DOI： 10.16451/j.cnki.issn1003-6059.202105009http://dx.doi.org/10.16451/j.cnki.issn1003-6059.202105009］
Zhou X， Guo G B， Sun Z and Liu Y. 2020. Multi-facet user preference learning for fine-grained item recommendation. Neurocomputing， 385： 258-268 ［DOI： 10.1016/j.neucom.2019.12.089http://dx.doi.org/10.1016/j.neucom.2019.12.089］
相关作者
相关机构
京公网安备11010802024621