Automatic texture exemplar extraction with jointed deep and broad learning models

Wu Huisi; Liang Chongxin; Yan Wei; Wen Zhenkun

doi:10.11834/jig.230184

Image Analysis and Recognition | Views : 0 下载量: 1167 CSCD: 0

PDF
Export
Share
Collection
Album

Automatic texture exemplar extraction with jointed deep and broad learning models
Vol. 29, Issue 4, Pages: 1003-1017(2024)
Received：14 April 2023，

Revised：2023-06-05，

Published：16 April 2024
DOI： 10.11834/jig.230184
稿件说明：

移动端阅览

吴惠思，梁崇鑫，颜威，文振焜. 2024. 联合深度学习和宽度学习的纹理样图自动提取. 中国图象图形学报， 29(04):1003-1017 DOI： 10.11834/jig.230184.

Wu Huisi， Liang Chongxin， Yan Wei， Wen Zhenkun. 2024. Automatic texture exemplar extraction with jointed deep and broad learning models. Journal of Image and Graphics， 29(04):1003-1017 DOI： 10.11834/jig.230184.

摘要

目的

纹理样图是指一幅用于描述纹理特征的图像，纹理样图多样性在纹理合成任务中是至关重要的，它可以为合成的纹理带来更丰富、多样和逼真的外观，同时为艺术家和设计师提供了更多的创作灵感和自由度。当前，纹理样图的提取主要通过手工剪裁和算法自动提取，从大量的图像中手工剪裁提取出高质量的纹理样图十分耗费精力和时间，并且该方式易受主观驱动且多样性受限。目前先进的纹理样图自动提取算法基于卷积神经网络的 Trimmed T-CNN（texture convolutional neural network）模型存在推理速度慢的问题。基于此，本文致力于利用互联网上丰富的图像资源，自动快速地从各种图像中裁剪出理想且多样的纹理样图，让用户有更多的选择。

方法

本文提出一个结合深度学习和宽度学习的从原始图像中自动提取纹理样图的方法。为了获取理想的纹理样图，首先通过残差特征金字塔网络提取特征图，有效地从输入图像中识别样图候选者，然后采用区域候选网络快速自动地获取大量的纹理样图候选区域。接下来，利用宽度学习系统对纹理样图的候选区域进行分类。最后，使用评分准则对宽度学习系统的分类结果进行评分，从而筛选出理想的纹理样图。

结果

为了验证本文方法的有效性，收集大量理想纹理样图并将它们分成6个类进行实验验证，本文模型的准确度达到了94.66%。与当前先进的方法Trimmed T-CNN相比，本文模型准确度提高了0.22%且速度得到了提升。对于分辨率为512 × 512像素、1 024 × 1 024像素和 2 048 × 2 048像素的图像，算法速度分别提快了1.393 8 s、1.864 3 s和2.368 7 s。

结论

本文提出的纹理样图自动提取算法，综合了深度学习和宽度学习的优点，使纹理样图的提取结果更加准确且高效。

Abstract

Objective

Texture exemplar refers to the input samples or templates for texture synthesis or generation that contains the desired texture features and structures. Texture synthesis refers to the generation of new texture images by combining or duplicating one or more texture samples. In the texture synthesis task based on the texture exemplar， the diversity and texture structure of the texture exemplar play a decisive role that determines the effect of the texture synthesis task. In the field of computer vision， texture sample diversity is crucial in texture synthesis tasks， which can bring richer， diverse， and realistic appearance to synthesized textures. Simultaneously， it can provide greater creative inspiration and design ideas to artists and designers. At present， texture exemplars can be extracted from multiple sources， such as public texture datasets， Internet picture clips， or photography. That is， texture exemplars are mostly extracted via manual cutting and automatic algorithm extraction. However， not everyone is an artist， and extracting a good texture sample or cutting out a small texture exemplar from an existing image is difficult for ordinary people. In addition， manually cropping and extracting high-quality texture samples from a large number of images consumes considerable energy and time for texture artists， and this method is easily driven by subjectivity and limited in diversity. With the development of deep learning algorithms， the currently used state-of-the-art automatic texture exemplar extraction algorithm is the Trimmed T-CNN model based on a convolutional neural network （CNN）. It can effectively extract a variety of texture exemplars from the input image. However， the model uses a selective search algorithm to generate a candidate region， and thus， this process is time-consuming and computationally complex， and the model suffers from slow inference speed. Considering the aforementioned reasons， this study is committed to using the rich image resources on the Internet to automatically， quickly， and accurately cut out ideal and diverse texture exemplars from various images， providing users with more choices， and to better meet the needs of texture synthesis task requirements.

Method

On the basis of the algorithm idea of object detection， we propose an automatic texture exemplar extraction algorithm that combines deep learning and broad learning. This algorithm generates candidate texture exemplar regions through CNN and then uses broad learning for classification. To obtain the ideal texture exemplar， this study first uses the residual feature pyramid network to extract feature maps from the original image， aiming to effectively identify and generate texture exemplar candidates from the input image and then using the region candidate network to automatically and quickly obtain a large number of multi-scale texture exemplar candidate regions. Subsequently， we leverage a broad learning system to classify the candidate regions of texture exemplars extracted in the previous step. Finally， to obtain the ideal texture exemplar， we designed a scoring criterion based on classification accuracy， distribution characteristics， and size， aiming to use the scoring criterion to score the classification results of the broad learning system to screen out the ideal texture exemplars.

Result

To verify the effectiveness of the proposed method， we first collect a large number of ideal texture exemplars with distinguishable and representative features as a training dataset and divide them into six classes based on size and regularity for experimental verification. A large number of qualitative and quantitative experiments are performed in this study. The experimental results show that the accuracy of the model developed in this study reaches 94.66%. Compared with the state-of-the-art method Trimmed T-CNN， the accuracy of the model in this study increases by 0.22% and speed is improved. In particular， for images with resolutions of 512 × 512 pixels， 1 024 × 1 024 pixels， and 2 048 × 2 048 pixels， the speed of the algorithm in this study is increased by 1.393 8 s， 1.864 3 s， and 2.368 7 s， respectively.

Conclusion

In this study， we propose an automatic texture exemplar extraction algorithm based on deep learning and broad learning. This algorithm effectively combines the advantages of CNNs and broad learning classification systems. The experimental results show that our model outperforms several state-of-the-art texture exemplar extraction methods， making texture exemplar extraction results more accurate and efficient.

关键词

Keywords

references

Andrearczyk V and Whelan P F . 2016 . Using filter banks in convolutional neural networks for texture classification . Pattern Recognition Letters ， 84 ： 63 - 69 ［ DOI： 10.1016/j.patrec.2016.08.016 http://dx.doi.org/10.1016/j.patrec.2016.08.016 ］

Caputo B ， Hayman E and Mallikarjuna P . 2005 . Class-specific material categorisation // Proceedings of the 10th IEEE International Conference on Computer Vision . Beijing， China ： IEEE： 1597 - 1604 ［ DOI： 10.1109/ICCV.2005.54 http://dx.doi.org/10.1109/ICCV.2005.54 ］

Chen C L P and Liu Z L . 2018 . Broad learning system： an effective and efficient incremental learning system without the need for deep architecture . IEEE Transactions on Neural Networks and Learning Systems ， 29 （ 1 ）： 10 - 24 ［ DOI： 10.1109/TNNLS.2017.2716952 http://dx.doi.org/10.1109/TNNLS.2017.2716952 ］

Cimpoi M ， Maji S ， Kokkinos I ， Mohamed S and Vedaldi A . 2014 . Describing textures in the wild // Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus， USA ： IEEE： 3606 - 3613 ［ DOI： 10.1109/CVPR.2014.461 http://dx.doi.org/10.1109/CVPR.2014.461 ］

Cimpoi M ， Maji S and Vedaldi A . 2015 . Deep filter banks for texture recognition and segmentation // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston， USA ： IEEE： 3828 - 3836 ［ DOI： 10.1109/CVPR.2015.7299007 http://dx.doi.org/10.1109/CVPR.2015.7299007 ］

Dai D X ， Riemenschneider H and Van Gool L . 2014 . The synthesizability of texture examples // Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus， USA ： IEEE： 3027 - 3034 ［ DOI： 10.1109/CVPR.2014.387 http://dx.doi.org/10.1109/CVPR.2014.387 ］

Hayman E ， Caputo B ， Fritz M and Eklundh J O . 2004 . On the significance of real-world conditions for material classification // Proceedings of the 8th European Conference on Computer Vision . Prague， Czech Republic ： Springer： 253 - 266 ［ DOI： 10.1007/978-3-540-24673-2_21 http://dx.doi.org/10.1007/978-3-540-24673-2_21 ］

He K M ， Zhang X Y ， Ren S Q and Sun J . 2016 . Deep residual learning for image recognition // Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas， USA ： IEEE： 770 - 778 ［ DOI： 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ］

Igelnik B and Pao Y H . 1995 . Stochastic choice of basis functions in adaptive function approximation and the functional-link net . IEEE Transactions on Neural Networks ， 6 （ 6 ）： 1320 - 1329 ［ DOI： 10.1109/72.471375 http://dx.doi.org/10.1109/72.471375 ］

Krizhevsky A ， Sutskever I and Hinton G E . 2017 . ImageNet classification with deep convolutional neural networks . Communications of the ACM ， 60 （ 6 ）： 84 - 90 ［ DOI： 10.1145/3065386 http://dx.doi.org/10.1145/3065386 ］

Kylberg G. 2011 . Kylberg texture dataset v . 1 .0. Uppsala ： Centre for Image Analysis， Swedish University of Agricultural Sciences and Uppsala University

Lazebnik S ， Schmid C and Ponce J . 2005 . A sparse texture representation using local affine regions . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 27 （ 8 ）： 1265 - 1278 ［ DOI： 10.1109/TPAMI.2005.151 http://dx.doi.org/10.1109/TPAMI.2005.151 ］

Lin T Y ， Doll􀅡r P ， Girshick R B ， He K M ， Hariharan B and Belongie S J . 2017 . Feature pyramid networks for object detection // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 936 - 944 ［ DOI： 10.1109/CVPR.2017.106 http://dx.doi.org/10.1109/CVPR.2017.106 ］

Lockerman Y ， Rushmeier H and Dorsey J . 2015 . Systems and methods for creating texture exemplars . U . S .， No. 900 ， 737 ， 3 B 2

Neubeck A and van Gool L . 2006 . Efficient non-maximum suppression // Proceedings of the 18th International Conference on Pattern Recognition . Hong Kong， China ： IEEE： 850 - 855 ［ DOI： 10.1109/ICPR.2006.479 http://dx.doi.org/10.1109/ICPR.2006.479 ］

Ni Z ， Cao G T and Duan Y . 2022 . Semi-supervised representative region texture extraction of façade ［EB/OL］. ［ 2023-03-30 ］. https://arxiv.org/pdf/2212.02220.pdf https://arxiv.org/pdf/2212.02220.pdf

Pao Y H ， Park G H and Sobajic D J . 1994 . Learning and generalization characteristics of the random vector functional-link net . Neurocomputing ， 6 （ 2 ）： 163 - 180 ［ DOI： 10.1016/0925-2312（94）90053-1 http://dx.doi.org/10.1016/0925-2312（94）90053-1 ］

Pao Y H and Takefuji Y . 1992 . Functional-link net computing： theory， system architecture， and functionalities . Computer ， 25 （ 5 ）： 76 - 79 ［ DOI： 10.1109/2.144401 http://dx.doi.org/10.1109/2.144401 ］

Ren S Q ， He K M ， Girshick R and Sun J . 2017 . Faster R-CNN： towards real-time object detection with region proposal networks . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 39 （ 6 ）： 1137 - 1149 ［ DOI： 10.1109/TPAMI.2016.2577031 http://dx.doi.org/10.1109/TPAMI.2016.2577031 ］

Sharan L ， Rosenholtz R and Adelson E . 2009 . Material perception： what can you see in a brief glance？ Journal of Vision ， 9 （ 8 ）： # 784 ［ DOI： 10.1167/9.8.784 http://dx.doi.org/10.1167/9.8.784 ］

Simonyan K and Zisserman A . 2015 . Very deep convolutional networks for large-scale image recognition // Proceedings of the 3rd International Conference on Learning Representations . San Diego， USA ： ICLR ［ DOI： 10.48550/arXiv.1409.1556 http://dx.doi.org/10.48550/arXiv.1409.1556 ］

Wang X H and Tao J Z . 2013 . Fast near-regular texture synthesis based on the concept of co-evolution . Journal of Image and Graphics ， 18 （ 1 ）： 107 - 114

王相海，陶兢喆 . 2013 . 协同进化的近似规则纹理合成 . 中国图象图形学报， 18 （ 1 ）： 107 - 114 ［ DOI： 10.11834/jig.20130114 http://dx.doi.org/10.11834/jig.20130114 ］

Wei L Y ， Lefebvre S ， Kwatra V and Turk G . 2009 . State of the art in example-based texture synthesis // Proceedings of the 30th Annual Conference of the European Association for Computer Graphics . Munich， Germany ： Eurographics Association： 93 - 117 ［ DOI： 10.2312/egst.20091063 http://dx.doi.org/10.2312/egst.20091063 ］

Wu H A ， Lyu X M and Wen Z K . 2018 . Automatic texture exemplar extraction based on global and local textureness measures . Computational Visual Media ， 4 （ 2 ）： 173 - 184 ［ DOI： 10.1007/s41095-018-0106-z http://dx.doi.org/10.1007/s41095-018-0106-z ］

Wu H S ， Yan W ， Li P and Wen Z K . 2021 . Deep texture exemplar extraction based on trimmed T-CNN . IEEE Transactions on Multimedia ， 23 ： 4502 - 4514 ［ DOI： 10.1109/TMM.2020.3043130 http://dx.doi.org/10.1109/TMM.2020.3043130 ］

Xu Y ， Ji H and Fermüller C . 2009 . Viewpoint invariant texture description using fractal analysis . International Journal of Computer Vision ， 83 （ 1 ）： 85 - 100 ［ DOI： 10.1007/s11263-009-0220-6 http://dx.doi.org/10.1007/s11263-009-0220-6 ］

Zhang H ， Xue J and Dana K . 2017 . Deep TEN： texture encoding network // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 2896 - 2905 ［ DOI： 10.1109/CVPR.2017.309 http://dx.doi.org/10.1109/CVPR.2017.309 ］

Zhou Y ， Zhu Z ， Bai X ， Lischinski D ， Cohen-Or D and Huang H . 2018 . Non-stationary texture synthesis by adversarial expansion . ACM Transactions on Graphics ， 37 （ 4 ）： # 49 ［ DOI： 10.1145/3197517.3201285 http://dx.doi.org/10.1145/3197517.3201285 ］

Zhu W H and Wei B G . 2008 . The technology of sampled-based texture synthesis . Journal of Image and Graphics ， 13 （ 11 ）： 2063 - 2069

朱文浩，魏宝刚 . 2008 . 基于样本的纹理合成技术综述 . 中国图象图形学报， 13 （ 11 ）： 2063 - 2069 ［ DOI： 10.11834/jig.20081101 http://dx.doi.org/10.11834/jig.20081101 ］