浏览全部资源
扫码关注微信
纸质出版日期： 2022-12-16 ，
录用日期： 2021-12-21
移动端阅览
王晨, 吴国华, 姚晔, 任一支, 王秋华, 袁理锋. 深度学习汉字生成与字体风格迁移综述[J]. 中国图象图形学报, 2022,27(12):3415-3428.
Chen Wang, Guohua Wu, Ye Yao, Yizhi Ren, Qiuhua Wang, Lifeng Yuan. Review of Chinese characters generation and font transfer based on deep learning[J]. Journal of Image and Graphics, 2022,27(12):3415-3428.
汉字字体风格迁移旨在保证在语义内容不变的同时对汉字的字形作相应的转换。由于深度学习在图像风格迁移任务中表现出色，因此汉字生成可以从汉字图像入手，利用此技术实现汉字字体的转换，减少字体设计的人工干预，减轻字体设计的工作负担。然而，如何提高生成图像的质量仍是一个亟待解决的问题。本文首先系统梳理了当前汉字字体风格迁移的相关工作，将其分为3类，即基于卷积神经网络(convolutional neural network，CNN)、自编码器(auto-encoder，AE)和生成对抗网络(generative adversarial network，GAN)的汉字字体风格迁移方法。然后，对比分析了22种汉字字体风格迁移方法在数据集规模方面的需求和对不同字体类别转换的适用能力，并归纳了这些方法的特点，包括细化汉字图像特征、依赖预训练模型提取有效特征、支持去风格化等。同时，按照汉字部首检字表构造包含多种汉字字体的简繁体汉字图像数据集，并选取代表性的汉字字体风格迁移方法进行对比实验，实现源字体(仿宋)到目标字体(印刷体和手写体)的转换，展示并分析Rewrite2、zi2zi、TET-GAN(texture effects transfer GAN)和Unet-GAN等4种代表性汉字字体风格迁移方法的生成效果。最后，对该领域的现状和挑战进行总结，展望该领域未来发展方向。由于汉字具有数量庞大和风格多样的特性，因此基于深度学习的汉字生成与字体风格迁移技术还不够成熟。未来该领域将从融合汉字的风格化与去风格化为一体、有效提取汉字特征等方面进一步探索，使字体设计工作向更灵活、个性化的方向发展。
Deep learning technology is capable of image-style transfer tasks recently. The Chinese characters font transfer is focused on content preservation while the font attribute is converted. Thanks to the emerging deep learning
the workload of font design for Chinese characters can be alleviated effectively and the restrictions of human intervention are avoided as well. However
the quality of generated images is still a challenging issue to be resolved. Our review is aimed at the analysis of the most representative image generation and font transfer methods for Chinese characters. The literature review of contemporary font transfer methods for Chinese characters is systematically summarized and divided into three categories: 1) convolutional neural network based (CNN-based)
2) auto-encoder based (AE-based)
and 3) generative adversarial networks based (GAN-based). To avoid information missing in the process of data reconstruction
a convolutional neural network extracted features of images without changing the dimensions of data. Auto-encoder processed the data through a deep neural network to learn the distribution of real samples and generate realistic fake samples. Generative adversarial networks became popular in Chinese characters font transfer after being proposed by Goodfellow. Its structure consists of a generator and a discriminator generally. The core idea of generative adversarial networks came from the Nash equilibrium of game theory
which is reflected in the process of continuous optimization between the generator and discriminator. Its generator learned the distribution of real data
generated fake images
and induced discriminators to make wrong decisions. The discriminator tried to determine whether the input data is real or fake. Through this game between generator and discriminator
the latter could not distinguish the real image from the fake in the end. According to the way of learning font style features of Chinese characters
we divided these methods based on GAN into three categories: 1) self-learning font style features
2) external font style features
and 3) extractive font style features. We introduced twenty-two font transfer methods for Chinese characters and summarized the performance of these methods in terms of dataset requirements
font category supports
and evaluations for generated images. The key factors of these font transfer methods are introduced
compared
and analyzed
including refining Chinese characters features
relying on a pre-trained model for effective feature extraction
and supporting de-stylization. According to the uniformed table of radicals for Chinese characters
we built a data set consisting of 6 683 simplified and traditional characters in five fonts. To accomplish the transformation from source font (simfs.ttf) to target font (printed font and hand-written font)
comparative experiments are carried out on the same data set. The comparative analysis of four archetypal font transfer methods for Chinese characters (Rewrite2
zi2zi
TET-GAN
and Unet-GAN) are implemented. Our quantitative evaluation metrics are composed of root mean square error (RMSE) and Pixel-level accuracy (pix_acc)
and several generated results of each method for comparison were shown. The strokes of characters generated by Unet-GAN are the most complete and clear according to the subjective and objective evaluation metrics of generated images
which is competent for the transfer and generation of printing and handwriting font. At the same time
the methods named Rewrite2
zi2zi
and TET-GAN are more suitable for the font transfer task of printing characters
and their ability to generate strokes of Chinese characters needs to be improved. We summarized some challenging issues like blurred strokes of Chinese characters
immature methods of multi-domain transformation
and large-scale training data set applications. The future research direction can be further extended on the aspects of 1) integrating the stylization and de-stylization of Chinese characters
2) reducing the size of the data set
and 3) extracting features of Chinese characters more effectively. Furthermore
its potential can be associated with information hiding technology for document watermarking and embedding secret messages.
汉字字体风格迁移图像生成卷积神经网络(CNN)自编码器(AE)生成对抗网络(GAN)
Chinese characters font transferimage generationconvolutional neural network (CNN)auto-encoder(AE)generate adversarial network (GAN)
Arjovsky M, Chintala S and Bottou L. 2017. Wasserstein GAN [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1701.07875.pdfhttps://arxiv.org/pdf/1701.07875.pdf
Chang B and Zhang Q. 2017. Rewrite2: a GAN based Chinese font transfer algorithm [EB/OL]. [2021-05-15].https://github.com/changebo/Rewrite2/https://github.com/changebo/Rewrite2/
Chang B, Zhang Q, Pan S Y and Meng L L. 2018a. Generating handwritten Chinese characters using CycleGAN//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, USA: IEEE: 199-207 [DOI: 10.1109/WACV.2018.00028http://dx.doi.org/10.1109/WACV.2018.00028]
Chang J. 2019. Automated Chinese Fonts Synthesis Based on Image Transformation. Shanghai: Shanghai Jiao Tong University
常杰. 2019. 基于图像转换的中文字体自动化生成. 上海: 上海交通大学
Chang J and Gu Y J. 2017. Chinese typography transfer [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1707.04904v1.pdfhttps://arxiv.org/pdf/1707.04904v1.pdf
Chang J, Gu Y J and Zhang Y. 2017. Chinese typeface transformation with hierarchical adversarial network [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1711.06448.pdfhttps://arxiv.org/pdf/1711.06448.pdf
Chang J, Gu Y J, Zhang Y and Wang Y F. 2018b. Chinese handwriting imitation with hierarchical generative adversarial network//Proceedings of British Machine Vision Conference. Newcastle, UK: BMVA Press: 1-12
Chen J F. 2020. Learning Chinese Character One-to-Many Style Transformation and Generation by Generative Adversarial Networks. Chengdu: University of Electronic Science and Technology of China
陈杰夫. 2020. 基于生成对抗网络的汉字字体风格生成与转换. 成都: 电子科技大学
Chen J F, Chen H, Xu X, Ji Y L and Chen L J. 2019a. Learning to write multi-stylized Chinese characters by generative adversarial networks. Journal of University of Electronic Science and Technology of China, 48(5): 674-678
陈杰夫, 陈华, 徐行, 姬艳丽, 陈李江. 2019a. 基于对抗生成网络的多风格化的汉字. 电子科技大学学报, 48(5): 674-678 [DOI: 10.3969/j.issn.1001-0548.2019.05.003]
Chen J F, Xu X, Ji Y L and Chen H. 2019b. Learning to create multi-stylized Chinese character fonts by generative adversarial networks//Proceedings of the ACM Turing Celebration Conference-China. New York, USA: Association for Computing Machinery: 1-6 [DOI: 10.1145/3321408.3322631http://dx.doi.org/10.1145/3321408.3322631]
Choi Y, Choi M, Kim M, Ha J W, Kim S and Choo J. 2018. StarGAN: unified generative adversarial networks for multi-domain image-to-image translation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 8789-8797 [DOI: 10.1109/CVPR.2018.00916http://dx.doi.org/10.1109/CVPR.2018.00916]
Gao Y, Guo Y, Lian Z H, Tang Y M and Xiao J G. 2019. Artistic glyph image synthesis via one-stage few-shot learning. ACM Transactions on Graphics, 38(6): 1-12 [DOI: 10.1145/3355089.3356574]
Gao Y M and Wu J Q. 2020. GAN-based unpaired Chinese character image translation via skeleton transformation and stroke rendering//Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press: 646-653 [DOI: 10.1609/aaai.v34i01.5405http://dx.doi.org/10.1609/aaai.v34i01.5405]
Gatys L A, Ecker A S and Bethge M. 2016. A neural algorithm of artistic style. Journal of Vision, 16(12): #326 [DOI: 10.1167/16.12.326]
Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A C and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press: 2672-2680 [DOI: 10.5555/2969033.2969125http://dx.doi.org/10.5555/2969033.2969125]
Isola P, Zhu J Y, Zhou T H and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5967-5976 [DOI: 10.1109/CVPR.2017.632http://dx.doi.org/10.1109/CVPR.2017.632]
Jiang Y, Lian Z H, Tang Y M and Xiao J G. 2017. DCFont: an end-to-end deep Chinese font generation system//Proceedings of SIGGRAPH Asia 2017 Technical Briefs. Bangkok, Thailand: Association for Computing Machinery: #22 [DOI: 10.1145/3145749.3149440http://dx.doi.org/10.1145/3145749.3149440]
Kingma D P and Welling M. 2014. Auto-Encoding variational Bayes [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1312.6114.pdfhttps://arxiv.org/pdf/1312.6114.pdf
Lei Y T, Zhou L G, Pan T J, Qian H H and Sun Z L. 2018. Learning and generation of personal handwriting style Chinese font//Proceedings of 2018 IEEE International Conference on Robotics and Biomimetics. Kuala Lumpur, Malaysia: IEEE: 1909-1914 [DOI: 10.1109/ROBIO.2018.8665297http://dx.doi.org/10.1109/ROBIO.2018.8665297]
Li M, Wang J, Yang Y, Huang W X and Du W J. 2019. Improving GAN-based calligraphy character generation using graph matching//Proceedings of the 19th International Conference on Software Quality, Reliability and Security Companion. Sofia, Bulgaria: IEEE: 291-295 [DOI: 10.1109/QRS-C.2019.00062http://dx.doi.org/10.1109/QRS-C.2019.00062]
Liu X Y, Meng G F, Chang J L, Hu R G, Xiang S M and Pan C H. 2021. Decoupled representation learning for character glyph synthesis. IEEE Transactions on Multimedia, 24: 1787-1799 [DOI: 10.1109/TMM.2021.3072449]
Lu S Y and Hsiang T R. 2018. Generating Chinese typographic and handwriting fonts from a small font sample set//Proceedings of 2018 International Joint Conference on Neural Networks. Rio de Janeiro, Brazil: IEEE: 1-8 [DOI: 10.1109/IJCNN.2018.8489500http://dx.doi.org/10.1109/IJCNN.2018.8489500]
Lyu P Y, Bai X, Yao C, Zhu Z, Huang T T and Liu W Y. 2017. Auto-encoder guided GAN for Chinese calligraphy synthesis//Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition. Kyoto, Japan: IEEE: 1095-1100 [DOI: 10.1109/ICDAR.2017.181http://dx.doi.org/10.1109/ICDAR.2017.181]
Ma Y F, Dong Y T, Li K, Ren J, Geng G H and Zhou M Q. 2019. A survey of Chinese character style transfer//Wang Y T, Huang Q M, Peng Y X, eds. Image and Graphics Technologies and Applications (IGTA). Singapore, Singapore: Springer: 392-404 [DOI: 10.1007/978-981-13-9917-6_38http://dx.doi.org/10.1007/978-981-13-9917-6_38]
Mirza M and Osindero S. 2014. Conditional generative adversarial nets [EB/OL]. [2021-11-06].https://arxiv.org/pdf/1411.1784.pdfhttps://arxiv.org/pdf/1411.1784.pdf
Odena A, Olah C and Shlens J. 2016. Conditional image synthesis with auxiliary classifier GANs [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1610.09585.pdfhttps://arxiv.org/pdf/1610.09585.pdf
Perarnau G, Van De Weijer J, Raducanu B andálvarez J M. 2016. Invertible conditional GANs for image editing [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1611.06355.pdfhttps://arxiv.org/pdf/1611.06355.pdf
Ren C Y, Lyu S J, Zhan H J and Lu Y. 2019. SAFont: automatic font synthesis using self-Attention mechanisms. Australia Journal of Intelligent Information Processing Systems, 2019, 16(2): 19-25
Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241 [DOI: 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28]
Shi D Q. 2018. Generative Adversarial Network a Primer. Beijing: China Machine Press
史丹青. 2018. 生成对抗网络入门指南. 北京: 机械工业出版社
Sun D H, Zhang Q and Yang J. 2018a. Pyramid embedded generative adversarial network for automated font generation//Proceedings of the 24th International Conference on Pattern Recognition, Beijing, China: IEEE: 976-981 [DOI: 10.1109/ICPR.2018.8545701http://dx.doi.org/10.1109/ICPR.2018.8545701]
Sun D Y, Ren T Z, Li C X, Su H and Zhu J. 2018b. Learning to write stylized Chinese characters by reading a handful of examples//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm, Sweden: International Joint Conferences on Artificial Intelligence Organization: 920-927 [DOI: 10.24963/ijcai.2018/128http://dx.doi.org/10.24963/ijcai.2018/128]
Sun S, Zhang W M, Fang H and Yu N H. 2022. Automatic generation of Chinese document watermarking fonts. Journal of Image and Graphics, 27(1): 262-276
孙杉, 张卫明, 方涵, 俞能海. 2022. 中文水印字库的自动生成方法. 中国图象图形学报, 27(1): 262-276 [DOI: 10.11834/jig.200695]
Teng S H and Kong L R. 2019. Chinese fonts style transfer based on generative adversarial networks. Application Research of Computers, 36(10): 3164-3167
滕少华, 孔棱睿. 2019. 基于生成式对抗网络的中文字体风格迁移. 计算机应用研究, 36(10): 3164-3167 [DOI: 10.19734/j.issn.1001-3695.2018.04.0330]
Tian Y C. 2017. Rewrite: neural style transfer for Chinese fonts [EB/OL]. [2021-05-15].https://github.com/kaonashi-tyc/Rewrite/https://github.com/kaonashi-tyc/Rewrite/
Tian Y C and Chong Z. 2017. Zi2zi: master Chinese calligraphy with conditional adversarial networks [EB/OL]. [2021-05-15].https://github.com/kaonashi-tyc/zi2zi/https://github.com/kaonashi-tyc/zi2zi/
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł and Polosukhin I. 2017. Attention is all you need// Proceedings of the 31st Advances in Neural Information Processing Systems, Red Hook, USA: Curran Associates Inc.: 6000-6010
Wang C, Zhu Y N, Shen Z Y, Wang D, Wu G H and Yao Y. 2021. Font transfer based on parallel auto-encoder for glyph perturbation via strokes moving//Proceedings of International Conference on Algorithms and Architectures for Parallel Processing: the 21st International Conference. Berlin, Germany: Springer-Verlag: 586-602 [DOI: 10.1007/978-3-030-95388-1_39http://dx.doi.org/10.1007/978-3-030-95388-1_39]
Wu L, Zhang C Q, Liu J M, Han J Y, Liu J T, Ding E R and Bai X. 2019. Editing text in the wild//Proceedings of the 27th ACM International Conference on Multimedia. Nice, France: Association for Computing Machinery (ACM): 1500-1508 [DOI: 10.1145/3343031.3350929http://dx.doi.org/10.1145/3343031.3350929]
Wu S J, Yang C Y and Hsu J Y. 2020. CalliGAN: style and structure-aware Chinese calligraphy character generator [EB/OL]. [2021-05-15].https://arxiv.org/pdf/2005.12500.pdfhttps://arxiv.org/pdf/2005.12500.pdf
Xiao F X. 2018. The Style-Transfer of Chinese Character Based on Deep Learning. Guangzhou: Jinan University
肖奋溪. 2018. 基于深度学习的汉字字体风格迁移. 广州: 暨南大学
Xiao F X, Zhang J, Huang B and Wu X. 2020. Multiform fonts-to-fonts translation via style and content disentangled representations of Chinese character [EB/OL]. [2021-05-15].https://arxiv.org/pdf/2004.03338.pdfhttps://arxiv.org/pdf/2004.03338.pdf
Yang S, Liu J Y, Wang W J and Guo Z M. 2019. TET-GAN: text effects transfer via stylization and destylization//Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu, USA: AAAI Press: 1238-1245 [DOI: 10.1609/aaai.v33i01.33011238http://dx.doi.org/10.1609/aaai.v33i01.33011238]
Yu W M and D-miracleAce. 2018. Style migration for artistic font with CNN [EB/OL]. [2021-05-15].https://github.com/yuweiming70/Style_Migration_For_Artistic_Font_With_CNNhttps://github.com/yuweiming70/Style_Migration_For_Artistic_Font_With_CNN
Zeng J S, Chen Q, Liu Y X, Wang M W and Yao Y. 2020. StrokeGAN: reducing mode collapse in Chinese font generation via stroke encoding [EB/OL]. [2021-05-15].https://arxiv.org/pdf/2012.08687.pdfhttps://arxiv.org/pdf/2012.08687.pdf
Zhang J W, Chen D N, Han G Q, Li G Z, He J T, Liu Z M and Ruan Z H. 2020a. SSNet: structure-semantic net for Chinese typography generation based on image translation. Neurocomputing, 371: 15-26 [DOI: 10.1016/j.neucom.2019.08.072]
Zhang Y X, Zhang Y and Cai W B. 2018. Separating style and content for generalized style transfer//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 8447-8455 [DOI: 10.1109/CVPR.2018.00881http://dx.doi.org/10.1109/CVPR.2018.00881]
Zhang Y X, Zhang Y and Cai W B. 2020b. A unified framework for generalizable style transfer: style and content separation. IEEE Transactions on Image Processing, 29: 4085-4098 [DOI: 10.1109/TIP.2020.2969081]
Zhang Y Y. 2019. Generating Handwritten Chinese Characters with GANs. Shanghai: East China Normal University
张艺颖. 2019. 基于生成对抗网络的手写体汉字生成. 上海: 华东师范大学
Zheng Z Z and Zhang F Y. 2018. Coconditional autoencoding adversarial networks for Chinese font feature learning [EB/OL]. [2021-05-15].https://arxiv.org/pdf/1812.04451.pdfhttps://arxiv.org/pdf/1812.04451.pdf
Zhu A N, Lu X B, Bai X, Uchida S, Iwana B K and Xiong S W. 2020. Few-shot text style transfer via deep feature similarity. IEEE Transactions on Image Processing, 29: 6932-6946 [DOI: 10.1109/TIP.2020.2995062]
相关作者
相关机构
京公网安备11010802024621