浏览全部资源
扫码关注微信
Published： 16 December 2023 ，
移动端阅览
饶师瑾， 钱文华， 张结宝. 2023. 曲线笔触渲染的图像风格迁移. 中国图象图形学报， 28(12):3825-3837
Rao Shijin， Qian Wenhua， Zhang Jiebao. 2023. Image style transfer via curved stroke rendering. Journal of Image and Graphics， 28(12):3825-3837
目的
2
针对GANILLA、Paint Transformer、StrokeNet等已有的风格迁移算法存在生成图像笔触丢失、线条灵活度低以及训练时间长等问题，提出一种基于曲线笔触渲染的图像风格迁移算法。
方法
2
首先按照自定义的超像素数量将图像前景分割为小区域的子图像，保留更多图像细节，背景分割为较大区域的子图像，再对分割后的每个子区域选取控制点，采用Bezier方程对控制点进行多尺度笔触生成，最后采用风格迁移算法将渲染后的图像与风格图像进行风格迁移。
结果
2
与AST（arbitrary style transfer）方法相比，本文方法在欺骗率指标上提升了0.13，测试者欺骗率提升了0.13。与Paint Transformer等基于笔触渲染的算法对比，本文能够在纹理丰富的前景区域生成细粒度笔触，在背景区域生成粗粒度笔触，保存更多的图像细节。
结论
2
与GANILLA、AdaIN（adaptive instance normalization）等风格迁移算法相比，本文采用图像分割算法取点生成笔触参数，无需训练，不仅提高了算法效率，而且生成的多风格图像保留风格化图像的笔触绘制痕迹，图像色彩鲜明。
Objective
2
The goal of the image style transfer algorithm is to render the content of one image with the style of another image. Image style transfer methods can be divided into traditional and neural style transfer methods. Traditional style transfer methods can be broadly classified to stroke-based rendering （SBR） and image analogy （IA）. SBR simulates human drawings with different sizes of strokes. Meanwhile， the main idea of IA is as follows： given a pair of images
A
（unprocessed source image） and
A
′ （processed image） and the unprocessed image
B
， the processed image
B
′ is obtained by processing
B
in the same way as
A
to
A
′. Meanwhile， neural style transfer methods can be categorized into slow image reconstruction methods based on online image optimization and fast image reconstruction methods based on offline model optimization. Slow image reconstruction methods optimize the image in the pixel space and minimize the objective function via gradient descent. Using a random noise as the starting image， the pixel values of the noise images are iteratively changed to obtain a target result image. Given that each reconstruction result requires many iterative optimizations in the pixel space， this approach consumes much time and computational resources and requires a high time overhead. In order to speed up this process， fast image reconstruction methods are proposed to train the network in advance in a data-driven manner using a large amount of data. Given an input， the trained network only needs one forward transmission to output a style transfer image. In recent years， the seminal works on style transfer have focused on building a neural network that can effectively extract the content and style features of an image and then combine these features to generate highly realistic images. However， building a model for each style is inefficient and requires much labor and time resources. One example of this model is the neural style transfer （NST） algorithm， which aims at transferring the texture of the style image to the content image and optimizing the noise at the pixel level step by step. However， hand-painted paintings comprise different strokes that are made using different brush sizes and textures. Compared with human paintings， the NST algorithm only generates photo-realistic imageries and ignores paint strokes or stipples. Given that the existing style transfer algorithms， such as Ganilla and Paint Transformer， suffer from loss of brush strokes and poor stroke flexibility， we propose a novel style transfer algorithm to quickly recreate the content of one image with curved strokes and then transfer another style to the re-rendered image. The images generated using our method resemble those made by humans.
Method
2
First， we segment the content image into subregions with different scales via content mask according to the customized number of super pixels. Given that we do not pay attention to the background， we segment the image background into small subregions. To preserve large amounts of details， we save the image foreground as much as possible via segmentation into small subregions. The segmentations for the image foreground are twice greater than those for the image background. For each subregion， four control points are selected in the convex hull of a subregion， and then the Bezier equation is used to generate thick strokes in the background and thin strokes in the foreground. The image rendered with strokes is then stylized with the style image by using the style transfer algorithm to generate a stylized image that retains the stroke traces.
Result
2
Compared with the arbitrary style transfer （AST） and Kotovenko’s method， the deception rate of the proposed method is increased by 0.13 and 0.04， respectively， while its human deception rate is increased by 0.13 and 0.01.Compared with Paint Transformer and other stroke-based rendering algorithms， our proposed method can generate thin strokes in the texture-rich foreground region and thick strokes in the background， thus preserving large amounts of image details.
Conclusion
2
Unlike whitening and coloring transforms （WCT）， AdaIN， and other style transfer algorithms， the proposed method uses an image segmentation algorithm to generate stroke parameters without training， thus improving efficiency and generating multi-style images that preserve the stroke drawing traces of stylized images with vivid colors.
非真实感渲染风格迁移笔触渲染（SBR）Bezier曲线超像素分割
non-photorealistic renderingstyle transferstroke-based rendering（SBR）Bezier curvesuperpixel segmentation
Achanta R， Shaji A， Smith K， Lucchi A， Fua P and Süsstrunk S. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence， 34（11）： 2274-2282 ［DOI： 10.1109/TPAMI.2012.120http://dx.doi.org/10.1109/TPAMI.2012.120］
Amami A， Azouz Z B and Alouane M T H. 2019. AdaSLIC： adaptive supervoxel generation for volumetric medical images. Multimedia Tools and Applications， 78（3）： 3723-3745 ［DOI： 10.1007/s11042-017-5563-3http://dx.doi.org/10.1007/s11042-017-5563-3］
An J， Huang S Y， Song Y B， Dou D J， Liu W and Luo J B. 2021. ArtFlow： unbiased image style transfer via reversible neural flows//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 862-871 ［DOI： 10.1109/CVPR46437.2021.00092http://dx.doi.org/10.1109/CVPR46437.2021.00092］
Ganin Y， Kulkarni T， Babuschin I， Ali Eslami S M and Vinyals O. 2018. Synthesizing programs for images using reinforced adversarial learning ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1804.01118.pdfhttps://arxiv.org/pdf/1804.01118.pdf
Gatys L A， Ecker A S and Bethge M. 2015. A neural algorithm of artistic style ［EB/OL］. ［2015-08-26］. https://arxiv.org/pdf/1508.06576.pdfhttps://arxiv.org/pdf/1508.06576.pdf
Gatys L A， Ecker A S and Bethge M. 2016. Image style transfer using convolutional neural networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 2414-2423 ［DOI： 10.1109/CVPR.2016.265http://dx.doi.org/10.1109/CVPR.2016.265］
Ghariba B， Shehata M S and McGuire P. 2022. Salient object detection using semantic segmentation technique. International Journal of Computational Vision and Robotics， 12（1）： 17-38 ［DOI： 10.1504/IJCVR.2022.119240http://dx.doi.org/10.1504/IJCVR.2022.119240］
Hertzmann A. 1998. Painterly rendering with curved brush strokes of multiple sizes//Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. ［s.l.］： ACM： 453-460 ［DOI： 10.1145/280814.280951http://dx.doi.org/10.1145/280814.280951］
Hertzmann A. 2003. A survey of stroke-based rendering. IEEE Computer Graphics and Applications， 23（4）： 70-81 ［DOI： 10.1109/MCG.2003.1210867http://dx.doi.org/10.1109/MCG.2003.1210867］
Hertzmann A， Jacobs C E， Oliver N， Curless B and Salesin D H. 2001. Image analogies//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ［s.l.］： ACM： 327-340 ［DOI： 10.1145/383259.383295http://dx.doi.org/10.1145/383259.383295］
Hicsonmez S， Samet N， Akbas E and Duygulu P. 2020. Ganilla： generative adversarial networks for image to illustration translation. Image and vision computing， 95： #103886 ［DOI：10.1016/j.imavis.2020.103886http://dx.doi.org/10.1016/j.imavis.2020.103886］
Huang X and Belongie S. 2017. Arbitrary style transfer in real-time with adaptive instance normalization//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 1510-1519 ［DOI： 10.1109/ICCV.2017.167http://dx.doi.org/10.1109/ICCV.2017.167］
Huang Z W， Zhou S C and Heng W. 2019. Learning to paint with model-based deep reinforcement learning//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision （ICCV）. Seoul， Korea （South）： IEEE： 8708-8717 ［DOI： 10.1109/ICCV.2019.00880http://dx.doi.org/10.1109/ICCV.2019.00880］
Irving B. 2016. maskSLIC： regional superpixel generation with application to local pathology characterisation in medical images ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1606.09518.pdfhttps://arxiv.org/pdf/1606.09518.pdf
Johnson J， Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision （ECCV）. Amsterdam， the Netherlands： Springer： 694-711 ［DOI： 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43］
Kotovenko D， Wright M， Heimbrecht A and Ommer B. 2021. Rethinking style transfer： from pixels to parameterized brushstrokes//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 12191-12200 ［DOI： 10.1109/CVPR46437.2021.01202http://dx.doi.org/10.1109/CVPR46437.2021.01202］
Li Y J， Fang C， Yang J M， Wang Z W， Lu X and Yang M H. 2017. Universal style transfer via feature transforms ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1705.08086.pdfhttps://arxiv.org/pdf/1705.08086.pdf
Lin T W， Ma Z Q， Li F， He D L， Li X， Ding E R， Wang N N， Li J and Gao X B. 2021. Drafting and revision： Laplacian pyramid network for fast high-quality artistic style transfer//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Nashville， USA： IEEE： 5137-5146 ［DOI： 10.1109/CVPR46437.2021.00510http://dx.doi.org/10.1109/CVPR46437.2021.00510］
Lin T Y， Maire M， Belongie S， Hays J， Perona P， Ramanan D， Dollr P and Zitnick C l. 2014. Microsoft coco： common objects in context//Proceedings of 2014 European Conference on Computer Vision. Zurich， Switzerland： 740-755［DOI：10.1007/978-3-319-10602-1\_48］
Liu S H， Lin T W， He D L， Li F， Deng R F， Li X， Ding E R and Wang H. 2021. Paint transformer： feed forward neural painting with stroke prediction//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 6578-6587 ［DOI： 10.1109/ICCV48922.2021.00653http://dx.doi.org/10.1109/ICCV48922.2021.00653］
Liu Z L， Zhu W and Yuan Z Y. 2019. Image instance style transfer combined with fully convolutional network and cycleGAN. Journal of Image and Graphics， 24（8）： 1283-1291
刘哲良， 朱玮， 袁梓洋. 2019. 结合全卷积网络与CycleGAN的图像实例风格迁移. 中国图象图形学报， 24（8）： 1283-1291 ［DOI： 10.11834/jig.180624http://dx.doi.org/10.11834/jig.180624］
Nichol K. 2016. Painter by numbers， wikiart ［EB/OL］. ［2022-12-22］. https://www.kaggle.com/c/painter-by-numbershttps://www.kaggle.com/c/painter-by-numbers.
Sanakoyue A， Kotovenko D， Lang S and Ommer B. 2018. A style-aware content loss for real-time HD style transfer//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： IEEE： 715-731 ［DOI： 10.1007/978-3-030-01237-3_43http://dx.doi.org/10.1007/978-3-030-01237-3_43］
Song Y Z， Rosin P L， Hall P M and Collomosse J . 2008. Arty shapes//The 4th International Symposium on Computational Aesthetics in Graphics， Visualization， and Imaging. Lisbon， Portugal： 65-72［DOI： 10.2312/compaesth/compaesth08/065-072http://dx.doi.org/10.2312/compaesth/compaesth08/065-072］
Wang P， Li Y J and Vasconcelos N. 2021. Rethinking and improving the robustness of image style transfer//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 124-133 ［DOI： 10.1109/CVPR46437.2021.00019http://dx.doi.org/10.1109/CVPR46437.2021.00019］
Wang Z Z， Zhao L， Chen H B， Qiu L H， Mo Q H， Lin S H， Xing W and Lu D M. 2020. Diversified arbitrary style transfer via deep feature perturbation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 7786-7795 ［DOI： 10.1109/CVPR42600.2020.00781http://dx.doi.org/10.1109/CVPR42600.2020.00781］
Xie B， Wang N and Fan Y W. 2020. Correlation alignment total variation model and algorithm for style transfer. Journal of Image and Graphics， 25（2）： 241-254
谢斌， 汪宁， 范有伟. 2020. 相关对齐的总变分风格迁移新模型. 中国图象图形学报， 25（2）： 241-254 ［DOI： 10.11834/jig.190199http://dx.doi.org/10.11834/jig.190199］
Xie N， Hachiya H and Sugiyama M. 2013. Artist agent： a reinforcement learning approach to automatic stroke generation in oriental ink painting. IEICE Transactions on Information and Systems， E96.D（5）： 1134-1144 ［DOI： 10.1587/transinf.E96.D.1134http://dx.doi.org/10.1587/transinf.E96.D.1134］
Zhang Y L， Fang C， Wang Y L， Wang Z W， Lin Z， Fu Y and Yang J M. 2019. Multimodal style transfer via graph cuts//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 5942-5950 ［DOI： 10.1109/ICCV.2019.00604http://dx.doi.org/10.1109/ICCV.2019.00604］
Zheng N Y， Jiang Y F and Huang D J. 2023. StrokeNet： a neural painting environment ［EB/OL］. ［2022-12-22］. https://openreview.net/forum?id=HJxwDiActXhttps://openreview.net/forum?id=HJxwDiActX
Zou Z X， Shi T Y， Qiui S， Yuan Y and Shi Z W. 2021. Stylized neural painting//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Nashville， USA： IEEE： 15684-15693 ［DOI： 10.1109/CVPR46437.2021.01543http://dx.doi.org/10.1109/CVPR46437.2021.01543］
相关文章
相关作者
相关机构
京公网安备11010802024621