深度学习汉字生成与字体风格迁移综述

王晨; 吴国华; 姚晔; 任一支; 王秋华; 袁理锋

doi:10.11834/jig.210360

综述 | 浏览量 : 0 下载量: 3138 CSCD: 0

PDF
导出
分享
收藏
专辑

深度学习汉字生成与字体风格迁移综述
Review of Chinese characters generation and font transfer based on deep learning
2022年27卷第12期页码：3415-3428
收稿：2021-06-01，

修回：2021-12-14，

录用：2021-12-21，

纸质出版：2022-12-16
DOI： 10.11834/jig.210360
稿件说明：

移动端阅览

王晨, 吴国华, 姚晔, 任一支, 王秋华, 袁理锋. 深度学习汉字生成与字体风格迁移综述[J]. 中国图象图形学报, 2022,27(12):3415-3428. DOI： 10.11834/jig.210360.

Chen Wang, Guohua Wu, Ye Yao, Yizhi Ren, Qiuhua Wang, Lifeng Yuan. Review of Chinese characters generation and font transfer based on deep learning[J]. Journal of Image and Graphics, 2022, 27(12): 3415-3428. DOI： 10.11834/jig.210360.

摘要

汉字字体风格迁移旨在保证在语义内容不变的同时对汉字的字形作相应的转换。由于深度学习在图像风格迁移任务中表现出色，因此汉字生成可以从汉字图像入手，利用此技术实现汉字字体的转换，减少字体设计的人工干预，减轻字体设计的工作负担。然而，如何提高生成图像的质量仍是一个亟待解决的问题。本文首先系统梳理了当前汉字字体风格迁移的相关工作，将其分为3类，即基于卷积神经网络(convolutional neural network，CNN)、自编码器(auto-encoder，AE)和生成对抗网络(generative adversarial network，GAN)的汉字字体风格迁移方法。然后，对比分析了22种汉字字体风格迁移方法在数据集规模方面的需求和对不同字体类别转换的适用能力，并归纳了这些方法的特点，包括细化汉字图像特征、依赖预训练模型提取有效特征、支持去风格化等。同时，按照汉字部首检字表构造包含多种汉字字体的简繁体汉字图像数据集，并选取代表性的汉字字体风格迁移方法进行对比实验，实现源字体(仿宋)到目标字体(印刷体和手写体)的转换，展示并分析Rewrite2、zi2zi、TET-GAN(texture effects transfer GAN)和Unet-GAN等4种代表性汉字字体风格迁移方法的生成效果。最后，对该领域的现状和挑战进行总结，展望该领域未来发展方向。由于汉字具有数量庞大和风格多样的特性，因此基于深度学习的汉字生成与字体风格迁移技术还不够成熟。未来该领域将从融合汉字的风格化与去风格化为一体、有效提取汉字特征等方面进一步探索，使字体设计工作向更灵活、个性化的方向发展。

Abstract

Deep learning technology is capable of image-style transfer tasks recently. The Chinese characters font transfer is focused on content preservation while the font attribute is converted. Thanks to the emerging deep learning

the workload of font design for Chinese characters can be alleviated effectively and the restrictions of human intervention are avoided as well. However

the quality of generated images is still a challenging issue to be resolved. Our review is aimed at the analysis of the most representative image generation and font transfer methods for Chinese characters. The literature review of contemporary font transfer methods for Chinese characters is systematically summarized and divided into three categories: 1) convolutional neural network based (CNN-based)

2) auto-encoder based (AE-based)

and 3) generative adversarial networks based (GAN-based). To avoid information missing in the process of data reconstruction

a convolutional neural network extracted features of images without changing the dimensions of data. Auto-encoder processed the data through a deep neural network to learn the distribution of real samples and generate realistic fake samples. Generative adversarial networks became popular in Chinese characters font transfer after being proposed by Goodfellow. Its structure consists of a generator and a discriminator generally. The core idea of generative adversarial networks came from the Nash equilibrium of game theory

which is reflected in the process of continuous optimization between the generator and discriminator. Its generator learned the distribution of real data

generated fake images

and induced discriminators to make wrong decisions. The discriminator tried to determine whether the input data is real or fake. Through this game between generator and discriminator

the latter could not distinguish the real image from the fake in the end. According to the way of learning font style features of Chinese characters

we divided these methods based on GAN into three categories: 1) self-learning font style features

2) external font style features

and 3) extractive font style features. We introduced twenty-two font transfer methods for Chinese characters and summarized the performance of these methods in terms of dataset requirements

font category supports

and evaluations for generated images. The key factors of these font transfer methods are introduced

compared

and analyzed

including refining Chinese characters features

relying on a pre-trained model for effective feature extraction

and supporting de-stylization. According to the uniformed table of radicals for Chinese characters

we built a data set consisting of 6 683 simplified and traditional characters in five fonts. To accomplish the transformation from source font (simfs.ttf) to target font (printed font and hand-written font)

comparative experiments are carried out on the same data set. The comparative analysis of four archetypal font transfer methods for Chinese characters (Rewrite2

zi2zi

TET-GAN

and Unet-GAN) are implemented. Our quantitative evaluation metrics are composed of root mean square error (RMSE) and Pixel-level accuracy (pix_acc)

and several generated results of each method for comparison were shown. The strokes of characters generated by Unet-GAN are the most complete and clear according to the subjective and objective evaluation metrics of generated images

which is competent for the transfer and generation of printing and handwriting font. At the same time

the methods named Rewrite2

zi2zi

and TET-GAN are more suitable for the font transfer task of printing characters

and their ability to generate strokes of Chinese characters needs to be improved. We summarized some challenging issues like blurred strokes of Chinese characters

immature methods of multi-domain transformation

and large-scale training data set applications. The future research direction can be further extended on the aspects of 1) integrating the stylization and de-stylization of Chinese characters

2) reducing the size of the data set

and 3) extracting features of Chinese characters more effectively. Furthermore

its potential can be associated with information hiding technology for document watermarking and embedding secret messages.

关键词

Keywords

references

Arjovsky M, Chintala S and Bottou L. 2017. Wasserstein GAN [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1701.07875.pdf https://arxiv.org/pdf/1701.07875.pdf

Chang B and Zhang Q. 2017. Rewrite2: a GAN based Chinese font transfer algorithm [EB/OL]. [2021-05-15] . https://github.com/changebo/Rewrite2/ https://github.com/changebo/Rewrite2/

Chang B, Zhang Q, Pan S Y and Meng L L. 2018a. Generating handwritten Chinese characters using CycleGAN//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, USA: IEEE: 199-207 [ DOI: 10.1109/WACV.2018.00028 http://dx.doi.org/10.1109/WACV.2018.00028 ]

Chang J. 2019. Automated Chinese Fonts Synthesis Based on Image Transformation. Shanghai: Shanghai Jiao Tong University

常杰. 2019. 基于图像转换的中文字体自动化生成. 上海: 上海交通大学

Chang J and Gu Y J. 2017. Chinese typography transfer [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1707.04904v1.pdf https://arxiv.org/pdf/1707.04904v1.pdf

Chang J, Gu Y J and Zhang Y. 2017. Chinese typeface transformation with hierarchical adversarial network [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1711.06448.pdf https://arxiv.org/pdf/1711.06448.pdf

Chang J, Gu Y J, Zhang Y and Wang Y F. 2018b. Chinese handwriting imitation with hierarchical generative adversarial network//Proceedings of British Machine Vision Conference. Newcastle, UK: BMVA Press: 1-12

Chen J F. 2020. Learning Chinese Character One-to-Many Style Transformation and Generation by Generative Adversarial Networks. Chengdu: University of Electronic Science and Technology of China

陈杰夫. 2020. 基于生成对抗网络的汉字字体风格生成与转换. 成都: 电子科技大学

Chen J F, Chen H, Xu X, Ji Y L and Chen L J. 2019a. Learning to write multi-stylized Chinese characters by generative adversarial networks. Journal of University of Electronic Science and Technology of China, 48(5): 674-678

陈杰夫, 陈华, 徐行, 姬艳丽, 陈李江. 2019a. 基于对抗生成网络的多风格化的汉字. 电子科技大学学报, 48(5): 674-678 [DOI: 10.3969/j.issn.1001-0548.2019.05.003]

Chen J F, Xu X, Ji Y L and Chen H. 2019b. Learning to create multi-stylized Chinese character fonts by generative adversarial networks//Proceedings of the ACM Turing Celebration Conference-China. New York, USA: Association for Computing Machinery: 1-6 [ DOI: 10.1145/3321408.3322631 http://dx.doi.org/10.1145/3321408.3322631 ]

Choi Y, Choi M, Kim M, Ha J W, Kim S and Choo J. 2018. StarGAN: unified generative adversarial networks for multi-domain image-to-image translation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 8789-8797 [ DOI: 10.1109/CVPR.2018.00916 http://dx.doi.org/10.1109/CVPR.2018.00916 ]

Gao Y, Guo Y, Lian Z H, Tang Y M and Xiao J G. 2019. Artistic glyph image synthesis via one-stage few-shot learning. ACM Transactions on Graphics, 38(6): 1-12 [DOI: 10.1145/3355089.3356574]

Gao Y M and Wu J Q. 2020. GAN-based unpaired Chinese character image translation via skeleton transformation and stroke rendering//Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press: 646-653 [ DOI: 10.1609/aaai.v34i01.5405 http://dx.doi.org/10.1609/aaai.v34i01.5405 ]

Gatys L A, Ecker A S and Bethge M. 2016. A neural algorithm of artistic style. Journal of Vision, 16(12): #326 [DOI: 10.1167/16.12.326]

Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A C and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press: 2672-2680 [ DOI: 10.5555/2969033.2969125 http://dx.doi.org/10.5555/2969033.2969125 ]

Isola P, Zhu J Y, Zhou T H and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5967-5976 [ DOI: 10.1109/CVPR.2017.632 http://dx.doi.org/10.1109/CVPR.2017.632 ]

Jiang Y, Lian Z H, Tang Y M and Xiao J G. 2017. DCFont: an end-to-end deep Chinese font generation system//Proceedings of SIGGRAPH Asia 2017 Technical Briefs. Bangkok, Thailand: Association for Computing Machinery: #22 [ DOI: 10.1145/3145749.3149440 http://dx.doi.org/10.1145/3145749.3149440 ]

Kingma D P and Welling M. 2014. Auto-Encoding variational Bayes [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1312.6114.pdf https://arxiv.org/pdf/1312.6114.pdf

Lei Y T, Zhou L G, Pan T J, Qian H H and Sun Z L. 2018. Learning and generation of personal handwriting style Chinese font//Proceedings of 2018 IEEE International Conference on Robotics and Biomimetics. Kuala Lumpur, Malaysia: IEEE: 1909-1914 [ DOI: 10.1109/ROBIO.2018.8665297 http://dx.doi.org/10.1109/ROBIO.2018.8665297 ]

Li M, Wang J, Yang Y, Huang W X and Du W J. 2019. Improving GAN-based calligraphy character generation using graph matching//Proceedings of the 19th International Conference on Software Quality, Reliability and Security Companion. Sofia, Bulgaria: IEEE: 291-295 [ DOI: 10.1109/QRS-C.2019.00062 http://dx.doi.org/10.1109/QRS-C.2019.00062 ]

Liu X Y, Meng G F, Chang J L, Hu R G, Xiang S M and Pan C H. 2021. Decoupled representation learning for character glyph synthesis. IEEE Transactions on Multimedia, 24: 1787-1799 [DOI: 10.1109/TMM.2021.3072449]

Lu S Y and Hsiang T R. 2018. Generating Chinese typographic and handwriting fonts from a small font sample set//Proceedings of 2018 International Joint Conference on Neural Networks. Rio de Janeiro, Brazil: IEEE: 1-8 [ DOI: 10.1109/IJCNN.2018.8489500 http://dx.doi.org/10.1109/IJCNN.2018.8489500 ]

Lyu P Y, Bai X, Yao C, Zhu Z, Huang T T and Liu W Y. 2017. Auto-encoder guided GAN for Chinese calligraphy synthesis//Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition. Kyoto, Japan: IEEE: 1095-1100 [ DOI: 10.1109/ICDAR.2017.181 http://dx.doi.org/10.1109/ICDAR.2017.181 ]

Ma Y F, Dong Y T, Li K, Ren J, Geng G H and Zhou M Q. 2019. A survey of Chinese character style transfer//Wang Y T, Huang Q M, Peng Y X, eds. Image and Graphics Technologies and Applications (IGTA). Singapore, Singapore: Springer: 392-404 [ DOI: 10.1007/978-981-13-9917-6_38 http://dx.doi.org/10.1007/978-981-13-9917-6_38 ]

Mirza M and Osindero S. 2014. Conditional generative adversarial nets [EB/OL]. [2021-11-06] . https://arxiv.org/pdf/1411.1784.pdf https://arxiv.org/pdf/1411.1784.pdf

Odena A, Olah C and Shlens J. 2016. Conditional image synthesis with auxiliary classifier GANs [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1610.09585.pdf https://arxiv.org/pdf/1610.09585.pdf

Perarnau G, Van De Weijer J, Raducanu B andálvarez J M. 2016. Invertible conditional GANs for image editing [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1611.06355.pdf https://arxiv.org/pdf/1611.06355.pdf

Ren C Y, Lyu S J, Zhan H J and Lu Y. 2019. SAFont: automatic font synthesis using self-Attention mechanisms. Australia Journal of Intelligent Information Processing Systems, 2019, 16(2): 19-25

Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241 [ DOI: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]

Shi D Q. 2018. Generative Adversarial Network a Primer. Beijing: China Machine Press

史丹青. 2018. 生成对抗网络入门指南. 北京: 机械工业出版社

Sun D H, Zhang Q and Yang J. 2018a. Pyramid embedded generative adversarial network for automated font generation//Proceedings of the 24th International Conference on Pattern Recognition, Beijing, China: IEEE: 976-981 [ DOI: 10.1109/ICPR.2018.8545701 http://dx.doi.org/10.1109/ICPR.2018.8545701 ]

Sun D Y, Ren T Z, Li C X, Su H and Zhu J. 2018b. Learning to write stylized Chinese characters by reading a handful of examples//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm, Sweden: International Joint Conferences on Artificial Intelligence Organization: 920-927 [ DOI: 10.24963/ijcai.2018/128 http://dx.doi.org/10.24963/ijcai.2018/128 ]

Sun S, Zhang W M, Fang H and Yu N H. 2022. Automatic generation of Chinese document watermarking fonts. Journal of Image and Graphics, 27(1): 262-276

孙杉, 张卫明, 方涵, 俞能海. 2022. 中文水印字库的自动生成方法. 中国图象图形学报, 27(1): 262-276 [DOI: 10.11834/jig.200695]

Teng S H and Kong L R. 2019. Chinese fonts style transfer based on generative adversarial networks. Application Research of Computers, 36(10): 3164-3167

滕少华, 孔棱睿. 2019. 基于生成式对抗网络的中文字体风格迁移. 计算机应用研究, 36(10): 3164-3167 [DOI: 10.19734/j.issn.1001-3695.2018.04.0330]

Tian Y C. 2017. Rewrite: neural style transfer for Chinese fonts [EB/OL]. [2021-05-15] . https://github.com/kaonashi-tyc/Rewrite/ https://github.com/kaonashi-tyc/Rewrite/

Tian Y C and Chong Z. 2017. Zi2zi: master Chinese calligraphy with conditional adversarial networks [EB/OL]. [2021-05-15] . https://github.com/kaonashi-tyc/zi2zi/ https://github.com/kaonashi-tyc/zi2zi/

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł and Polosukhin I. 2017. Attention is all you need// Proceedings of the 31st Advances in Neural Information Processing Systems, Red Hook, USA: Curran Associates Inc.: 6000-6010

Wang C, Zhu Y N, Shen Z Y, Wang D, Wu G H and Yao Y. 2021. Font transfer based on parallel auto-encoder for glyph perturbation via strokes moving//Proceedings of International Conference on Algorithms and Architectures for Parallel Processing: the 21st International Conference. Berlin, Germany: Springer-Verlag: 586-602 [ DOI: 10.1007/978-3-030-95388-1_39 http://dx.doi.org/10.1007/978-3-030-95388-1_39 ]

Wu L, Zhang C Q, Liu J M, Han J Y, Liu J T, Ding E R and Bai X. 2019. Editing text in the wild//Proceedings of the 27th ACM International Conference on Multimedia. Nice, France: Association for Computing Machinery (ACM): 1500-1508 [ DOI: 10.1145/3343031.3350929 http://dx.doi.org/10.1145/3343031.3350929 ]

Wu S J, Yang C Y and Hsu J Y. 2020. CalliGAN: style and structure-aware Chinese calligraphy character generator [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/2005.12500.pdf https://arxiv.org/pdf/2005.12500.pdf

Xiao F X. 2018. The Style-Transfer of Chinese Character Based on Deep Learning. Guangzhou: Jinan University

肖奋溪. 2018. 基于深度学习的汉字字体风格迁移. 广州: 暨南大学

Xiao F X, Zhang J, Huang B and Wu X. 2020. Multiform fonts-to-fonts translation via style and content disentangled representations of Chinese character [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/2004.03338.pdf https://arxiv.org/pdf/2004.03338.pdf

Yang S, Liu J Y, Wang W J and Guo Z M. 2019. TET-GAN: text effects transfer via stylization and destylization//Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu, USA: AAAI Press: 1238-1245 [ DOI: 10.1609/aaai.v33i01.33011238 http://dx.doi.org/10.1609/aaai.v33i01.33011238 ]

Yu W M and D-miracleAce. 2018. Style migration for artistic font with CNN [EB/OL]. [2021-05-15] . https://github.com/yuweiming70/Style_Migration_For_Artistic_Font_With_CNN https://github.com/yuweiming70/Style_Migration_For_Artistic_Font_With_CNN

Zeng J S, Chen Q, Liu Y X, Wang M W and Yao Y. 2020. StrokeGAN: reducing mode collapse in Chinese font generation via stroke encoding [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/2012.08687.pdf https://arxiv.org/pdf/2012.08687.pdf

Zhang J W, Chen D N, Han G Q, Li G Z, He J T, Liu Z M and Ruan Z H. 2020a. SSNet: structure-semantic net for Chinese typography generation based on image translation. Neurocomputing, 371: 15-26 [DOI: 10.1016/j.neucom.2019.08.072]

Zhang Y X, Zhang Y and Cai W B. 2018. Separating style and content for generalized style transfer//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 8447-8455 [ DOI: 10.1109/CVPR.2018.00881 http://dx.doi.org/10.1109/CVPR.2018.00881 ]

Zhang Y X, Zhang Y and Cai W B. 2020b. A unified framework for generalizable style transfer: style and content separation. IEEE Transactions on Image Processing, 29: 4085-4098 [DOI: 10.1109/TIP.2020.2969081]

Zhang Y Y. 2019. Generating Handwritten Chinese Characters with GANs. Shanghai: East China Normal University

张艺颖. 2019. 基于生成对抗网络的手写体汉字生成. 上海: 华东师范大学

Zheng Z Z and Zhang F Y. 2018. Coconditional autoencoding adversarial networks for Chinese font feature learning [EB/OL]. [2021-05-15] . https://arxiv.org/pdf/1812.04451.pdf https://arxiv.org/pdf/1812.04451.pdf

Zhu A N, Lu X B, Bai X, Uchida S, Iwana B K and Xiong S W. 2020. Few-shot text style transfer via deep feature similarity. IEEE Transactions on Image Processing, 29: 6932-6946 [DOI: 10.1109/TIP.2020.2995062]