Gradual model reconstruction of Dongba painting based on residual dense structure

Mengjie Jiang; Wenhua Qian; Dan Xu; Hao Wu; Chunyu Liu

doi:10.11834/jig.200523

Image Processing and Coding | Views : 0 下载量: 0 CSCD: 1

PDF
Export
Share
Collection
Album

Gradual model reconstruction of Dongba painting based on residual dense structure
Vol. 27, Issue 4, Pages: 1084-1096(2022)
Published： 16 April 2022 ，

Accepted： 15 December 2020
DOI： 10.11834/jig.200523
稿件说明：

移动端阅览

Mengjie Jiang, Wenhua Qian, Dan Xu, Hao Wu, Chunyu Liu. Gradual model reconstruction of Dongba painting based on residual dense structure. [J]. Journal of Image and Graphics 27(4):1084-1096(2022)
DOI：

Mengjie Jiang, Wenhua Qian, Dan Xu, Hao Wu, Chunyu Liu. Gradual model reconstruction of Dongba painting based on residual dense structure. [J]. Journal of Image and Graphics 27(4):1084-1096(2022) DOI： 10.11834/jig.200523.

摘要

目的

东巴画具有内容丰富、线条疏密相间、色彩多样的独特艺术风格，将现有针对自然图像的超分辨率算法直接应用于低分辨率东巴画时，对东巴画线条、色块以及材质的重建效果不理想。为了有效提高东巴画数字图像分辨率，本文针对东巴画提出超分辨率重建方法。

方法

首先，针对东巴画图像包含丰富高频信息的特点搭建重建网络，网络整体结构采用多级子网络级联方式渐进地重建出高分辨率东巴画，多级子网络标签共同指导重建，减少了东巴画图像在上采样过程中高频细节的丢失，每一级子网络内部均包含浅层特征提取模块和以残差密集结构为核心的深层特征提取模块，分别提取东巴画不同层次的特征进行融合，改善了卷积层简单的链式堆叠造成的特征丢失。其次，为了进一步提升重建东巴画的视觉质量，在像素损失的基础上引入感知损失和对抗损失进行对抗训练。最后，为了使网络对东巴画图像特征的学习更具针对性，本文自建东巴画数据集用于网络训练。

结果

实验结果表明，在本文东巴画测试集set20上，当上采样因子为8时，相较于Bicubic(bicubic interpolation)、SRCNN(super-resolution convolutional neural network)、Srresnet(super-resolution residual network)和IMDN(information multi-distillation network)方法，本文算法的峰值信噪比分别增加了3.28 dB、1.80 dB、0.23 dB和0.36 dB，重建东巴画的主观视觉质量也得到了更好的结果。

结论

本文提出的超分辨率网络模型能有效提高低分辨率东巴画的分辨率和清晰度，并且具有普适性，亦可采用其他数据集进行训练以拓展应用范围。

Abstract

Objective

Dongba painting is an essential part of Na'xi culture in China. It is the fundamental medium to develop Dongba culture in art theory

natural aesthetics

religion and history. However

the current low resolution digital image of Dongba painting has affected the application

inheritance and development of Dongba culture. Super-resolution reconstruction technology is the process of recovering high-resolution image from low-resolution image. Because Dongba painting has a unique artistic style

compared with natural images

there are no dimension

distance and depth factors

and there is no light and shadow effect via natural light. While the existing super-resolution algorithm for natural images applied to Dongba painting straitforward

the reconstruction effect of lines

color blocks and materials of Dongba painting is not ideal. Therefore

the super-resolution reconstruction of Dongba painting is adopted and high-resolution Dongba painting obtained via the reconstruction of low-resolution Dongba painting.

Method

First

the Dongba painting dataset for network training is constructed

which makes the learning of Dongba painting image characteristics more targeted. The data set contains 298 high-definition Dongba paintings

and each Dongba painting has one dimension with a resolution of 2 K at least. Among them

278 paintings are used for training

20 paintings are used for testing

and the training set is randomly cropped for data enhancement. Next

in accordance with the characteristics of Dongba painting image which is rich in high frequency information

a reconstruction network is built and named Dongba super-resolution network (DBSRN): the overall structure of the network uses multi-level sub network cascade to gradually reconstruct high-resolution Dongba painting. During the feedforward process of a large-scale reconstruction

multiple intermediate Super-resolution predictions were produced. The final reconstruction results are constrained by multiple intermediate prediction values

so as to reconstruct Dongba paintings of different scales from small to large gradually. The reconstruction results of each sub network and the corresponding scale tags are calculated at the pixel level

and the tags of different scales jointly guide the reconstruction. The loss of high-frequency details is reduced in the up sampling process of Dongba painting image. To extract and fuse features at different levels

each level of super-resolution sub-network has a shallow feature extraction module

a deep feature extraction module

and a global feature fusion module. In this way

the disappearance of features with the deepening of network can be avoided. The extracted feature maps are input to the up-sampling module for reconstruction. The residual dense structure

which combines residual connection and dense connection

can effectively enhance feature reuse and slow down gradient disappearance. The demonstrated algorithm takes the residual dense structure as the core in deep feature extraction module

extracts the deep feature of Dongba painting for fusion

reduce the feature loss caused by simple chain stacking in the convolution layer; At the end

added-discriminator

perception loss and adversarial loss for adversarial training are implemented to improve the visual quality of Dongba painting based on pixel level loss. This network is named DBGAN (Dongba generative adversarial network).

Result

This illustration demonstrate the Dongba paintings reconstructed in this paper get better results in both subjective visual quality and objective indicators

compared to bicubic interpolation (Bicubic)

super-resolution convolutional neural network (SRCNN)

super-resolution residual network (Srresnet) and information multi-distillation network (IMDN).The peak signal to noise ratio (PSNR)/structural similarity index (SSIM) in DBSRN reach 33.46 dB and 0.911 2 when upsampling factor is 2

28.54 dB and 0.776 2 when upsampling factor is 4

and 24.61 dB and 0.643 0 when upsampling factor is 8. The objective indicators of DBSRN are improved to varying degrees compared with Bicubic

SRCNN

and Srresnet. Compared with Srresnet

when the upsampling factor is 2

PSNR and SSIM are increased by 0.10 dB and 0.000 8

respectively

when the upsampling factor is 4

they are increased by 0.18 dB and 0.003 2

and when the upsampling factor is 8

they are increased by 0.23 dB and 0.004 4. DBGAN improves the clarity and fidelity of the reconstruction results further and reconstruct Dongba paintings with more edge and texture details.

Conclusion

This super-resolution universal network model can improve the resolution and clarity of low-resolution Dongba paintings effectively.

关键词

东巴画超分辨率渐进式重建残差密集对抗训练

Keywords

Dongba paintingsuper-resolutionprogressive reconstructionresidual dense structureadversarial training

references

Dai D X, Wang Y J, Chen Y H and Van Gool L. 2016. Is image super-resolution helpful for other vision tasks?//Proceedings of 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Placid, USA: IEEE: 1-9[DOI: 10.1109/WACV.2016.7477613http://dx.doi.org/10.1109/WACV.2016.7477613]

Dong C, Loy C C, He K M and Tang X O. 2016a. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307[DOI: 10.1109/TPAMI.2015.2439281]

Dong C, Loy C C and Tang X O. 2016b. Accelerating the super-resolution convolutional neural network//Proceedings of the 16th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 391-407[DOI: 10.1007/978-3-319-46475-6_25http://dx.doi.org/10.1007/978-3-319-46475-6_25]

Feng R C, Gu J J, Qiao Y and Dong C. 2019. Suppressing model overfitting for image super-resolution networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach, USA: IEEE: 1964-1973[DOI: 10.1109/CVPRW.2019.00248http://dx.doi.org/10.1109/CVPRW.2019.00248]

Fu Y, Zhang T, Zheng Y Q, Zhang D B and Huang H. 2019. Hyperspectral image super-resolution with optimized RGB guidance//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 11653-11662[DOI: 10.1109/CVPR.2019.01193http://dx.doi.org/10.1109/CVPR.2019.01193]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 770-778[DOI: 10.1109/cvpr.2016.90http://dx.doi.org/10.1109/cvpr.2016.90]

Huang G, Liu Z, Van Der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 2261-2269[DOI: 10.1109/CVPR.2017.243http://dx.doi.org/10.1109/CVPR.2017.243]

Hui Z, Gao X B, Yang Y C and Wang X M. 2019. Lightweight image super-resolution with information multi-distillation network//Proceedings of the 27th ACM International Conference on Multimedia. Lisboa, Portugal: ACM: 2024-2032[DOI: 10.1145/3343031.3351084http://dx.doi.org/10.1145/3343031.3351084]

Isaac J S and Kulkarni R. 2015. Super resolution techniques for medical image processing//Proceedings of 2015 International Conference on Technologies for Sustainable Development (ICTSD). Mumbai, India: IEEE: 1-6[DOI: 10.1109/ICTSD.2015.7095900http://dx.doi.org/10.1109/ICTSD.2015.7095900]

Johnson J, Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 694-711[DOI: 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43]

Kim J, Lee J K and Lee K M. 2016. Accurate image super-resolution using very deep convolutional networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 1646-1654[DOI: 10.1109/CVPR.2016.182http://dx.doi.org/10.1109/CVPR.2016.182]

Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z H and Shi W Z. 2017. Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 105-114[DOI: 10.1109/CVPR.2017.19http://dx.doi.org/10.1109/CVPR.2017.19]

Li X and Orchard M T. 2001. New edge-directed interpolation. IEEE Transactions on Image Processing, 10(10): 1521-1527[DOI: 10.1109/83.951537]

Shi W Z, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D and Wang Z H. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 1874-1883[DOI: 10.1109/CVPR.2016.207http://dx.doi.org/10.1109/CVPR.2016.207]

Tong T, Li G, Liu X J and Gao Q Q. 2017. Image super-resolution using dense skip connections//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE: 4809-4817[DOI: 10.1109/ICCV.2017.514http://dx.doi.org/10.1109/ICCV.2017.514]

Wang H, Wu C D, Chi J N, Yu X S and Hu Q. 2020. Face super-resolution reconstruction based on multitask joint learning. Journal of Image and Graphics, 25(2): 229-240

王欢, 吴成东, 迟剑宁, 于晓升, 胡倩. 2020. 联合多任务学习的人脸超分辨率重建. 中国图象图形学报, 25(2): 229-240[DOI: 10.11834/jig.190233]

Wang X T, Yu K, Wu S X, Gu J J, Liu Y H, Dong C, Qiao Y and Loy C C. 2018. ESRGAN: enhanced super-resolution generative adversarial networks//Proceedings of 2018 European Conference on Computer Vision. Munich, Germany: Springer: 63-79[DOI: 10.1007/978-3-030-11021-5_5http://dx.doi.org/10.1007/978-3-030-11021-5_5]

Zhang Y L, Tian Y P, Kong Y, Zhong B N and Fu Y. 2018. Residual dense network for image super-resolution//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2472-2481[DOI: 10.1109/CVPR.2018.00262http://dx.doi.org/10.1109/CVPR.2018.00262]

Zhao H, Gallo O, Frosio I and Kautz J. 2017. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3(1): 47-57[DOI: 10.1109/TCI.2016.2644865]

Zhou D W, Zhao L J, Duan R and Chai X L. 2019. Image super-resolution based on recursive residual networks. Acta Automatica Sinica, 45(6): 1157-1165

周登文, 赵丽娟, 段然, 柴晓亮. 2019. 基于递归残差网络的图像超分辨率重建. 自动化学报, 45(6): 1157-1165[DOI: 10.16383/j.aas.c180334]

Alert me when the article has been cited

提交

Research progress on speech deepfake and its detection techniques

Deep learning-based spectral image super-resolution： a survey

Generalized adversarial defense against unseen attacks： a survey

Reversible image hiding with super-resolution behavior camouflage effect

Joint self-attention Transformer for multispectral and hyperspectral image fusion