浏览全部资源
扫码关注微信
纸质出版日期： 2020-07-16 ，
录用日期： 2020-02-19
移动端阅览
钱文华, 曹进德, 徐丹, 吴昊. 非真实感绘制技术研究现状与展望[J]. 中国图象图形学报, 2020,25(7):1283-1295.
Wenhua Qian, Jinde Cao, Dan Xu, Hao Wu. Research status and progress of non-photorealistic rendering technology[J]. Journal of Image and Graphics, 2020,25(7):1283-1295.
非真实感绘制技术（non-photorealistic rendering，NPR）主要用于模拟艺术风格、表现艺术特质和传达用户情感等，是计算机图形学的重要组成部分，其研究对象逐渐丰富，研究方法不断创新。本文从基于图像建模的绘制方法、基于深度学习的绘制方法、中国特有艺术作品的数字化模拟、非真实感情感特征识别以及非真实感视频场景绘制等5个方面概述目前研究进展，然后从扩展非真实感研究对象、增强视频绘制帧间连贯性、提取艺术风格情感特征以及评价非真实感绘制结果等4个角度讨论需要进一步研究的问题。针对需要深入研究的问题，指出提高算法的通用性和绘制效率，以及提高深度学习网络的泛化性，有助于扩展研究对象，模拟艺术风格的多样性，同时减小视频场景的帧间跳变；对艺术风格作品具有的情感特征、内在机理特征进行模拟，有助于提高绘制结果与艺术风格图像的相似度；结合主观和客观评价模型，可以更准确地对绘制结果进行评价，同时有利于优化网络模型参数，提高绘制效率。非真实感绘制在计算机视觉、文化遗产保护等领域具有重要的应用前景，但其研究对象、绘制算法、绘制效率仍然存在很多亟待解决的问题，随着硬件设备的不断改进，综合运用学科交叉知识、扩展应用领域将进一步推动非真实感绘制技术的发展。
Many art works have high artistic and commodity values. The production of realistic art works requires considerable time and strict manual skills. With the development of modern science and technology
the use of computer algorithms to synthesize and simulate different art styles is a popular topic in computer vision and image processing. Non-photorealistic rendering (NPR) technique mainly simulates art styles and expresses artistic characteristics. This technique is crucial for the protection and inheritance of art culture. Researchers have simulated and synthesized cartoon
sketch
oil painting
Chinese fine brushwork
calligraphy
and other art works on the basis of traditional image models. The development of deep learning has enabled researchers to use convolutional neural networks to transfer art features from style images to content images. In recent years
NPR research objects have been gradually enriched
research methods have been continuously enhanced
and the application fields of NPR have been expanding widely. However
the simulation and transfer of different styles using computers are challenges and open questions in NPR. This study reviews some NPR progresses in five important aspects
namely
traditional modeling rendering methods
depth learning transfer methods
digital simulation of Chinese unique art works
emotion recognition of art works
and artistic transfer from videos. We summarize the important stylized transfer methods in these aspects and show some simulation results. This paper further summarizes the issues that can be additional study fields in NPR
including research objects
video coherence among different frames
artistic style emotion
and evaluation criteria. Different art styles exist in the world
and the way to simulate these various works remain an open question. Although the transformation of video styles enriches the application field of NPR
it is a problem when the texture is complex
occlusion occurs
or the motion vector is inaccurate. Some artistic details are lost
which may cause discontinuity among video frames. NPR should extract high-level semantic features of art works in addition to simulating low-level artistic style features
such as color and texture. It can effectively improve the accuracy and efficiency of intelligent information retrieval and simulate
protect
and inherit artistic style through high-level semantic feature recognition and transfer. Evaluation standard also remains a problem. Whether the NPR algorithm can simulate real manual traces and users' creative intention has become the standard to evaluate the merits and demerits of the methods. This paper elaborates and concludes the problems that need to be further studied. First
we conclude that improving the generality and rendering efficiency of algorithms can expand research objects and simulate different art styles. Improved NPR algorithms can also raise the coherence among different video frames. Second
the recognition and extraction of emotional and intrinsic mechanism features from different stylistic works are another research fields. We can transfer the emotional and intrinsic features to the simulation results
which can improve the similarity between the results and artistic styles. Finally
the evaluation criteria for simulation results are an important research question. This paper concludes that combination with subjective and objective evaluation models can increase the evaluation accuracy of the results
and the combination can be applied to optimize the parameters of deep learning networks. NPR has important application prospects in computer vision
cultural heritage protection
education
and image processing. However
numerous problems remain to be solved. With the continuous improvement of hardware equipment
the comprehensive application of multidisciplinary knowledge and the expansion of application fields will promote the development of NPR.
非真实感绘制图像建模数字化模拟情感识别评价标准
non-photorealistic rendering(NPR)image modelingdigital simulationemotion recognitionevaluation criteria
Cao Y. 2012. Research on Image Based Rendering of Chinese Ink Painting. Changchun:Jilin University
曹毅. 2012.基于图像的中国水墨画绘制方法的研究.长春:吉林大学
Chen D D, Liao J, Yuan L, Yu N H and Hua G. 2017b. Coherent online video style transfer//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy, IEEE: 1114-1123[DOI: 10.1109/ICCV.2017.126http://dx.doi.org/10.1109/ICCV.2017.126]
Chen D D, Yuan L, Liao J, Yu N H and Hua G. 2017a. Stylebank: an explicit representation for neural image style transfer//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 2770-2779[DOI: 10.1109/CVPR.2017.296http://dx.doi.org/10.1109/CVPR.2017.296]
Chen T Q and Schmidt M. 2016. Fast patch-based style transfer of arbitrary style[EB/OL].[2019-12-15].https://arxiv.org/pdf/1612.04337.pdfhttps://arxiv.org/pdf/1612.04337.pdf
Chen Y, Lai Y K and Liu Y J. 2018. CartoonGAN: generative adversarial networks for photo cartoonization//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE: 9465-9474[DOI: 10.1109/CVPR.2018.00986http://dx.doi.org/10.1109/CVPR.2018.00986]
Dang K, Song G L and Ming A L. 2016. Chinese paper-cut generated from human portrait//Proceedings of 2016 IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW). Nantou, China: IEEE: 1-2[DOI: 10.1109/ICCE-TW.2016.7520943http://dx.doi.org/10.1109/ICCE-TW.2016.7520943]
Delanoy J, Bousseau A and Hertzmann A. 2019b. Video motion stylization by 2D rigidification//Proceedings of the 8th ACM/EG Expressive Symposium.[s.n.]: The Eurographics Association: 11-19[DOI: 10.2312/exp.20191072http://dx.doi.org/10.2312/exp.20191072]
Delanoy J, CoeurJolly D, Lachaud J O and Bousseau A. 2019a. Combining voxel and normal predictions for multi-view 3D sketching. Computers and Graphics, 82:65-72[DOI:10.1016/j.cag.2019.05.024]
Ekmekcioglu E, Velisavljević V and Worrall S T. 2011. Content adaptive enhancement of multi-view depth maps for free viewpoint video. Journal of Selected Topics in Signal Processing, 5(2):352-361[DOI:10.1109/jstsp.2010.2052783]
Fang J W, Yu J H and Zhang J S. 2012. Model and animation of water in Chinese painting style. Journal of Computer-Aided Design and Computer Graphics, 24(7):864-870
方建文, 于金辉, 张俊松. 2012.国画风格水动画建模.计算机辅助设计与图形学学报, 24(7):864-870)[DOI:10.3969/j.issn.1003-9775.2012.07.004]
Frigo O, Sabater N, Delon J and Hellier P. 2019. Video style transfer by consistent adaptive patch sampling. The Visual Computer, 35(3):429-443[DOI:10.1007/s00371-018-1474-1]
Gatys L A, Ecker A S and Bethge M. 2016. Image style transfer using convolutional neural networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 2414-2423[DOI: 10.1109/CVPR.2016.265http://dx.doi.org/10.1109/CVPR.2016.265]
Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2014. Generative adversarial networks//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press: 2672-2680
Guay M, Ronfard R, Gleicher M and Cani M P. 2015. Space-time sketching of character animation. ACM Transactions on Graphics, 34(4):1-10[DOI:10.1145/2766893]
Haseyama M, Ogawa T and Yagi N. 2013. A review of video retrieval based on image and video semantic understanding. ITE Transactions on Media Technology and Applications, 1(1):2-9[DOI:10.3169/mta.1.2]
Hertzmann A. 1998. Painterly rendering with curved brush strokes of multiple sizes//Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM: 453-460[DOI: 10.1145/280814.280951http://dx.doi.org/10.1145/280814.280951]
Iarussi E, Bommes D and Bousseau A. 2015. Bendfields:regularized curvature fields from rough concept sketches. ACM Transactions on Graphics, 34(3):1-16[DOI:10.1145/2710026]
Inglis T C, Inglis S and Kaplan C S. 2012. Op art rendering with lines and curves. Computers and Graphics, 36(6):607-621[DOI:10.1016/j.cag.2012.03.003]
Jia B, Brandt J, Mech R, Kim B and Manocha D. 2019. LPaintB: learning to paint from self-supervision[EB/OL].[2019-12-15].https://arxiv.org/pdf/1906.06841.pdfhttps://arxiv.org/pdf/1906.06841.pdf
Justin J, Alexandre A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 694-711[DOI: 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43]
Kyprianidis J E, Collomosse J, Wang T H and Isenberg T. 2013. State of the "art":a taxonomy of artistic stylization techniques for images and video. IEEE Transactions on Visualization and Computer Graphics, 19(5):866-885[DOI:10.1109/TVCG.2012.160]
Li B, Xiong C M, Wu T F, Zhou Y, Zhang L and Chu R F. 2018. Neural abstract style transfer for Chinese traditional painting//Proceedings of the 14th Asian Conference on Computer Vision. Perth, Australia: Springer: 212-227[DOI: 10.1007/978-3-030-20890-5_14http://dx.doi.org/10.1007/978-3-030-20890-5_14]
Li S K and Jiang Y B. 2013. Semi-supervised sentiment classification based on sentiment feature clustering. Journal of Computer Research and Development, 50(12):2570-2577
李素科, 蒋严冰. 2013.基于情感特征聚类的半监督情感分类.计算机研究与发展, 50(12):2570-2577)[DOI:10.1587/transinf.E96.D.2805]
Li W and Zhou C L. 2013. Automatic creation of artistic Chinese calligraphy. Journal of Software, 8(12):3048-3054[DOI:10.4304/jsw.8.12.3048-3054]
Li X Y, Zhang W, Shen T and Mei T. 2019b. Everyone is a cartoonist: selfie cartoonization with attentive adversarial networks//Proceedings of 2019 IEEE International Conference on Multimedia, Expo. Shanghai: IEEE: 652-657[DOI: 10.1109/ICME.2019.00118http://dx.doi.org/10.1109/ICME.2019.00118]
Li Y J, Fang C, Hertzmann A, Shechtman E and Yang M H. 2019a. Im2Pencil: controllable pencil illustration from photographs//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE: 1525-1534[DOI: 10.1109/CVPR.2019.00162http://dx.doi.org/10.1109/CVPR.2019.00162]
Lin Y L, Dai X Y, Li L, Wang X and Wang F Y. 2018. The new frontier of AI research:generative adversarial networks. Acta Automatica Sinica, 44(5):775-792
林懿伦, 戴星原, 李力, 王晓, 王飞跃. 2018.人工智能研究的新前线:生成式对抗网络.自动化学报, 44(5):775-792)[DOI:10.16383/j.aas.2018.y000002]
Liu J Y. 2013. Emotional Metaphor and Impressions——Dialysis Jim Dean's Portrait Print Art. Wuhan:Central China Normal University
刘金阳. 2013.情感隐喻与印痕——透析吉姆·戴恩肖像版画艺术.武汉:华中师范大学)[DOI:10.7666/d.Y2353412]
Liu Y S. 2015. Research on Improved Algorithm of Emotion Mapping of Print Based on LFCM-SVM. Kunming:Yunnan University
刘颖霜. 2015.基于LFCM-SVM的版画情感映射改进算法研究.昆明:云南大学
Lu C W, Xu L and Jia J Y. 2012. Combining sketch and tone for pencil drawing production//Proceedings of Symposium on Non-Photorealistic Animation and Rendering.[s.l.]: The Eurographics Association: 65-73[DOI: 10.2312/PE/NPAR/NPAR12/065-073http://dx.doi.org/10.2312/PE/NPAR/NPAR12/065-073]
Luan F J, Paris S, Shechtman E and Bala K. 2017. Deep photo style transfer//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA, IEEE: 6997-7005[DOI: 10.1109/CVPR.2017.740http://dx.doi.org/10.1109/CVPR.2017.740]
Montesdeoca S E, Seah H S, Bénard P, Vergne R, Thollot J, Rall H M and Benvenuti D. 2017a. Edge-and substrate-based effects for watercolor stylization//Symposium on Non-Photorealistic Animation and Rendering. New York: ACM: 1-10[DOI: 10.1145/3092919.3092928http://dx.doi.org/10.1145/3092919.3092928]
Montesdeoca S E, Seah H S, Rall H M and Benvenuti D. 2017b. Art-directed watercolor stylization of 3D animations in real-time. Computers and Graphics, 65:60-72[DOI:10.1016/j.cag.2017.03.002]
Nikulin Y and Novak R. 2016. Exploring the neural algorithm of artistic style[EB/OL].[2019-12-15].https://arxiv.org/pdf/1602.07188.pdfhttps://arxiv.org/pdf/1602.07188.pdf
Pu Y Y, Li W W, Zhao Z P, Qian W H and Xu D. 2013. Local color transfer algorithm based on CTWC and the application in styled rendering of Yunnan heavy color painting. Advances in Information Sciences&Service Science, 5(6):183-190[DOI:10.1016/j.egypro.2011.10.446]
Qian W H, Xu D, Cao J D, Guan Z and Pu Y Y. 2019. Aesthetic art simulation for embroidery style. Multimedia Tools and Applications, 78(1):995-1016[DOI:10.1007/s11042-018-6002-9]
Qian W H, Xu D, Yue K, Guan Z and Pu Y Y. 2013. Rendering pyrography style painting based on deviation mapping. Journal of Image and Graphics, 18(7):836-843
钱文华, 徐丹, 岳昆, 官铮, 普园媛. 2013.偏离映射的烙画风格绘制.中国图象图形学报, 18(7):836-843)[DOI:10.11834/jig.20130713]
Qian W H, Xu D, Yue K, Guan Z and Pu Y Y. 2015. Sketch artistic rendering based on significance map. Journal of Computer-Aided Design and Computer Graphics, 27(5):915-923
钱文华, 徐丹, 岳昆, 官铮, 普园媛. 2015.重要度引导的抽象艺术风格绘制.计算机辅助设计与图形学学报, 27(5):915-923)[DOI:10.3969/j.issn.1003-9775.2015.05.018]
Ruder M, Dosovitskiy A and Brox T. 2018. Artistic style transfer for videos and spherical images. International Journal of Computer Vision, 126(11):1199-1219[DOI:10.1007/s11263-018-1089-z]
Semmo A, Limberger D, Kyprianidis J E and Dollner J. 2016. Image stylization by interactive oil paint filtering. Computers and Graphics, 55:157-171[DOI:10.1016/j.cag.2015.12.001]
Sheng K K, Dong W M, Kong Y, Mei X, Li J L, Wang C J, Huang F Y and Hu B G. 2015. Evaluating the quality of face alignment without ground truth. Computer Graphics Forum, 34(7):213-223[DOI:10.1111/cgf.12760]
Spicker M, Kratt J, Arellano D and Deussen O. 2015. Depth-aware coherent line drawings//Proceedings of SIGGRAPH Asia 2015 Technical Briefs. New York: [s.n.]: 1-5[DOI: 10.1145/2820903.2820909http://dx.doi.org/10.1145/2820903.2820909]
Tang Y, Zhang Y, Shi X Y and Fan J. 2015. Multi-style video stylization based on texture advection. Science China Information Sciences, 58(11):1-13[DOI:10.1007/s11432-014-5255-9]
Ulyanov D, Lebedev V, Vedaldi A and Lempitsky V. 2016. Texture networks: feed-forward synthesis of textures and stylized images//Proceedings of the 33rd International Conference on Machine Learning. New York: JMLR: 1349-1357
Wang H Q and Hu L. 2017. Image emotional semantic classification and retrieval research. Journal of Chongqing Institute of Technology (Natural Science), 31(10):180-186
王华秋, 胡立. 2017.图像情感语义分类及检索研究.重庆理工大学学报(自然科学版), 31(10):180-186)[DOI:10.3969/j.issn.1674-8425(z).2017.10.029]
Wang X S, Li J, Hou J X, Xu D and Pu Y Y. 2015. Interactive digital synthesis of Yunnan out-of-print woodcuts. Journal of Image and Graphics, 20(7):937-944
王雪松, 李杰, 侯剑侠, 徐丹, 普园媛. 2015.交互式云南绝版套色木刻数字化合成.中国图象图形学报, 20(7):937-944)[DOI:10.11834/jig.20150710]
Xiang J H, Yang K W, Zhou J and Sun Z X. 2013. A novel image disintegration-based computerized embroidery method for random stitch. Journal of Graphics, 34(4):16-23
项建华, 杨克微, 周杰, 孙正兴. 2013.一种基于图像分解的乱针绣模拟生成方法.图学学报, 34(4):16-23)[DOI:10.3969/j.issn.2095-302X.2013.04.003]
Xing L and Tang D. 2017. Synthesis method for facial makeup of Peking opera paper cuts. Computer Engineering, 43(12):222-226
邢璐, 唐棣. 2017.京剧脸谱剪纸合成方法.计算机工程, 43(12):222-226)[DOI:10.3969/j.issn.1000-3428.2017.12.040]
Yang G Q. 2015. The Simulation of Brush Behavior Based on Force Feedback Technology. Dalian:Dalian University of Technology
杨广卿. 2015.基于力反馈技术的毛笔行为仿真研究.大连:大连理工大学
Yang F, Xia G S, Dai D X and Zhang L P. 2018. Learning the synthesizability of dynamic texture samples. IEEE Transactions on Image Processing, 28(5):2502-2517[DOI:10.1109/TIP.2018.2886807]
Yi R, Liu Y J, Lai Y K and Rosin P L. 2019. APDrawingGAN: generating artistic portrait drawings from face photos with hierarchical GANs//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE: 10735-10744[DOI: 10.1109/CVPR.2019.01100http://dx.doi.org/10.1109/CVPR.2019.01100]
Yin C, Gui Y, Xie Z F and Ma L Z. 2011. Shape context based video texture synthesis from still images//Proceedings of 2011 International Conference on Computational and Information Sciences. Chengdu, China: IEEE: 38-42[DOI: 10.1109/ICCIS.2011.252http://dx.doi.org/10.1109/ICCIS.2011.252]
Yu Y T, Xu D and Qian W H. 2019. Simulation of batik cracks and cloth dying. Scientia Sinica Informationis, 49(2):159-171
喻扬涛, 徐丹, 钱文华. 2019.蜡染冰纹生成与染色模拟.中国科学:信息科学, 49(2):159-171)[DOI:10.1360/N112018-00212]
Zhang J W, Wang R Z and Xu D. 2017. Automatic genaration of sketch-like pencil drawing from image//Proceedings of 2017 IEEE International Conference on Multimedia and Expo Workshops. Hong Kong, China: IEEE: 261-266[DOI: 10.1109/ICMEW.2017.8026301http://dx.doi.org/10.1109/ICMEW.2017.8026301]
Zhao J C. 2013. ComputerizedLearning and Classification of Traditional Chinese IWPs (Ink and Wash Paintings). Tianjin:Tianjin University
赵家川. 2013.中国传统水墨画的计算机分类与算法研究.天津:天津大学
Zhao P P. 2016. Research on the Technologies of Non-Photorealistic Ink Painting based on Image. Guilin:Guilin University of Electronic Technology
赵朋朋. 2016.基于图像的非真实感水墨画绘制技术研究.桂林:桂林电子科技大学
Zheng R, Qian W H, Xu D and Pu Y Y. 2019. Synthesis of embroidery based on convolutional neural network. Journal of Zhejiang University (Science Edition), 46(3):270-278
郑锐, 钱文华, 徐丹, 普园媛. 2019.基于卷积神经网络的刺绣风格数字合成.浙江大学学报(理学版), 46(3):270-278)[DOI:10.3785/j.issn.1008-9497.2019.03.002]
Zhou J, Sun Z X, Yang K W and Hu Z Z. 2014. Parametric generation method for irregular needling embroidery rendering. Journal of Computer -Aided Design and Computer Graphics, 26(3):436-444
周杰, 孙正兴, 杨克微, 胡宗正. 2014.乱针绣模拟的参数化生成方法.计算机辅助设计与图形学学报, 26(3):436-444)[DOI:10.3969/j.issn.1003-9775.2014.03.014]
Zhou X and Cao L. 2015. The sketch face recognition combining with AdaBoost and blocking LBP. Journal of Image and Graphics, 20(1):50-58
周汐, 曹林. 2015.分块LBP的素描人脸识别.中国图象图形学报, 20(1):50-58)[DOI:10.11834/jig.20150106]
Zhu J, Pu Y Y, Xu D, Qian W H and Wang L Q. 2017b. The performance analysis of low-resolution paintings for computational aesthetics//Proceedings of the 13th International Conference on Smart Graphics. Chengdu, China: Springer: 145-154[DOI: 10.1007/978-3-319-53838-9_12http://dx.doi.org/10.1007/978-3-319-53838-9_12]
Zhu J Y, Park T, Isola P and Efros A A. 2017a. Unpaired image-to-image translation using cycle-consistent adversarial networks[EB/OL].[2019-12-15].https://arxiv.org/pdf/1703.10593.pdfhttps://arxiv.org/pdf/1703.10593.pdf
Zitnick C L, Vedantam R and Parikh D. 2016. Adopting abstract images for semantic scene understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4):627-638[DOI:10.1109/TPAMI.2014.2366143]
京公网安备11010802024621