浏览全部资源
扫码关注微信
Published： 16 May 2022 ，
Accepted： 21 January 2021
移动端阅览
Xiangdan Hou, Haoran Liu, Hongpu Liu. High-resolution damaged images restoration based on convolutional auto-encoder generative adversarial network. [J]. Journal of Image and Graphics 27(5):1645-1656(2022)
目的
2
破损图像修复是一项具有挑战性的任务，其目的是根据破损图像中已知内容对破损区域进行填充。许多基于深度学习的破损图像修复方法对大面积破损的图像修复效果欠佳，且对高分辨率破损图像修复的研究也较少。对此，本文提出基于卷积自编码生成式对抗网络(convolutional auto-encoder generative adversarial network，CAE-GAN)的修复方法。
方法
2
通过训练生成器学习从高斯噪声到低维特征矩阵的映射关系，再将生成器生成的特征矩阵升维成高分辨率图像，搜索与待修复图像完好部分相似的生成图像，并将对应部分覆盖到破损图像上，实现高分辨率破损图像的修复。
结果
2
通过将学习难度较大的映射关系进行拆分，降低了单个映射关系的学习难度，提升了模型训练效果，在4个数据集上对不同破损程度的512×512×3高分辨率破损图像进行修复，结果表明，本文方法成功预测了大面积缺失区域的信息。与CE(context-encoders)方法相比，本文方法在破损面积大的图像上的修复效果提升显著，峰值信噪比(peak signal to noise ratio，PSNR)和结构相似性(structural similarity，SSIM)值最高分别提升了31.6%和18.0%，与DCGAN(deep convolutional generative adversarial network)方法相比，本文方法修复的图像内容符合度更高，破损区域修复结果更加清晰，PSNR和SSIM值最高分别提升了24.4%和50.0%。
结论
2
本文方法更适用于大面积破损图像与高分辨率图像的修复工作。
Objective
2
The integrity of information transmission can be achieved if the image is intact currently. However
the required image files are often damaged or obscured
such as the damage of old photos
and the obscuration of the required content in the surveillance image. The purpose of damaged images restoration is to fill the damaged part in terms of the recognized region in the damaged image. The regular method of image restoration inserts the damaged area in accordance with the surrounding information based on texture synthesis technology linearly. Although this type of method can repair the texture
it lacks the manipulation of the global structure and image semantics of the damaged image. Deep-learning-based damaged image restoration methods have been illustrated via the classical context-encoders model. Although this method can perform better restoration on the color and content of the damaged image
the effect on the detail texture restoration is not ideal
and the restoration result appears blurred. When the damaged area is large
the repair effect is not qualified due to the lack of available information. Simultaneously
there are fewer analyses on high-resolution damaged image restoration now. Most of the existing damaged image restoration experiments use 128×128×3 and smaller images
and there are fewer experiments to repair 512×512×3 and larger images. In order to solve the two problems of large-area damaged image repair and high-resolution image repair
this analysis demonstrates a restoration method based on convolutional auto-encoder generative adversarial network (CAE-GAN).
Method
2
The generator is trained to learn the mapping relationship from Gaussian noise to the low-dimensional feature matrix
and then the generated feature matrix is upgraded to a high-resolution image
and the generated image similar to the intact part of the image to be repaired is sorted out. The corresponding part restoration on the damaged image to complete the repair of the high-resolution damaged image. First
high-resolution images are encoded and then decoded via the convolutional auto-encoder training part. Then
the parameters fix of the convolutional auto-encoder is adopted to assist in training the adversarial generation network part. The generator can generate different codes based on random Gaussian noise and then be decoded into high-resolution images based on the trained decoder. At the end
an overall connected network training for search can generate appropriate noise. After the noise is up-sampled by the generator and decoder
it will output a generated image similar to the image to be repaired
and cover the corresponding part on the damaged image
and then realize the repair of high-resolution damaged images.
Result
2
By segmenting the mapping relationships that are difficult to learn
the learning barriers of a single mapping relationship is declined
and the model training effect is improved. The repair experiments are conducted on the CelebA dataset
the street view house number(SVHN) dataset
the Oxford 102 flowers dataset
and the Stanford cars dataset. This demonstration illustrates that the method predicts the information of a large area of missing areas in a good way. Compared with the context-encoders(CE) method
the method improves the restoration effect on images with large damaged areas significantly. The content of the repaired damaged area is closer to the related intact parts
and the texture connection is smoother. The peak signal to noise ratio (PSNR) value can be increased to 31.6%
and the structural similarity (SSIM) value can be increased to 18.0%. The PSNR value can be increased by 24.4%
and the SSIM value can be increased by 50.0%.
Conclusion
2
Therefore
the method is suitable for the image restoration based on large-area damaged images and high-resolution images.
破损图像修复高分辨率生成式对抗网络(GAN)大面积破损深度学习
damaged image repairhigh resolutiongenerative adversarial networks(GAN)large area damagedeep learning
Badrinarayanan V, Kendall A and Cipolla R. 2017. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12): 2481-2495[DOI: 10.1109/TPAMI.2016.2644615]
Barnes C, Shechtman E, Finkelstein A and Goldman D B. 2009. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 28(3): #24[DOI: 10.1145/1531326.1531330]
Criminisi A, Perez P and Toyama K. 2003. Object removal by exemplar-based inpainting//Proceedings of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Madison, USA: IEEE: #1211538[DOI: 10.1109/CVPR.2003.1211538http://dx.doi.org/10.1109/CVPR.2003.1211538]
Denton E, Chintala S, Szlam A and Fergus R. 2015. Deep generative image models using a Laplacian pyramid of adversarial networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press: 1486-1494
Efros A A and Freeman W T. 2001. Image quilting for texture synthesis and transfer//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA: ACM: 341-346[DOI: 10.1145/383259.383296http://dx.doi.org/10.1145/383259.383296]
Goodfellow I. 2017. NIPS 2016 tutorial: generative adversarial networks[EB/OL]. [2020-09-21].https://arxiv.org/pdf/1701.00160.pdfhttps://arxiv.org/pdf/1701.00160.pdf
Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press: 2672-2680
Hays J and Efros A A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics, 26(3): #1276377-1276382[DOI: 10.1145/1276377.1276382]
Heusel M, Ramsauer H, Unterthiner T, Nessler B and Hochreiter S. 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, USA: Curran Associates Inc. : 6629-6640
Huynh-Thu Q and Ghanbari M. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics Letters, 44(13): 800-801[DOI: 10.1049/el:20080522]
Iizuka S, Simo-Serra E and Ishikawa H. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics, 36(4): #107[DOI: 10.1145/3072959.3073659]
Johnson J, Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 694-711[DOI: 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43]
Kingma D P and Ba J. 2017. Adam: a method for stochastic optimization[EB/OL]. [2020-09-21].https://arxiv.org/pdf/1412.6980.pdfhttps://arxiv.org/pdf/1412.6980.pdf
Komodakis N. 2006. Image completion using global optimization//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York, USA: IEEE: 442-452[DOI: 10.1109/CVPR.2006.141http://dx.doi.org/10.1109/CVPR.2006.141]
Krause J, Stark M, Deng J and Li F F. 2013. 3D object representations for fine-grained categorization//Proceedings of 2013 IEEE International Conference on Computer Vision Workshops. Sydney, Australia: IEEE: 554-561[DOI: 10.1109/ICCVW.2013.77http://dx.doi.org/10.1109/ICCVW.2013.77]
Lecun Y, Bottou L, Bengio Y and Haffner P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): 2278-2324[DOI: 10.1109/5.726791]
Li Y J, Liu S F, Yang J M and Yang M H. 2017. Generative face completion//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 3911-3919[DOI: 10.1109/CVPR.2017.624http://dx.doi.org/10.1109/CVPR.2017.624]
Liu Z W, Luo P, Wang X G and Tang X O. 2015. Deep learning face attributes in the wild//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 3730-3738[DOI: 10.1109/ICCV.2015.425http://dx.doi.org/10.1109/ICCV.2015.425]
Masci J, Meier U, Cireşan D and Schmidhuber J. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction//Proceedings of the 21st International Conference on Artificial Neural Networks. Espoo, Finland: Springer: 52-59[DOI: 10.1007/978-3-642-21735-7_7http://dx.doi.org/10.1007/978-3-642-21735-7_7]
Nazeri K, Ng E, Joseph T, Qureshi F and Ebrahimi M. 2019. EdgeConnect: generative image inpainting with adversarial edge learning[EB/OL]. [2020-09-21]. https://arxiv.org/pdf/1901.00212.pdf
Netzer Y, Wang T, Coates A, Bissacco A, Wu B and Ng A Y. 2011. Reading digits in natural images with unsupervised feature learning//Proceedings of NIPS Workshop on Deep Learning and Unsupervised Feature Learning. Granada, Spain: NIPS: 12-17
Pathak D, Krähenbühl P, Donahue J, Darrell T and Efros A A. 2016. Context encoders: feature learning by inpainting//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 2536-2544[DOI: 10.1109/CVPR.2016.278http://dx.doi.org/10.1109/CVPR.2016.278]
Radford A, Metz L and Chintala S. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL]. [2020-09-21]. https://arxiv.org/pdf/1511.06434v1.pdf
Rumelhart D E, Hinton G E and Williams R J. 1986. Learning representations by back-propagating errors. Nature, 323(6088): 533-536[DOI: 10.1038/323533a0]
Song Y H, Yang C, Lin Z, Liu X F, Huang Q, Li H and Kuo C C J. 2018. Contextual-based image inpainting: infer, match, and translate//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 3-18[DOI: 10.1007/978-3-030-01216-8_1http://dx.doi.org/10.1007/978-3-030-01216-8_1]
Wang J Y, Zhou W G, Tang J H, Fu Z Q, Tian Q and Li H Q. 2018. Unregularized auto-encoder with generative adversarial networks for image generation//Proceedings of the 26th ACM International Conference on Multimedia. Seoul, Korea (South): ACM: 709-717[DOI: 10.1145/3240508.3240569http://dx.doi.org/10.1145/3240508.3240569]
Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612[DOI: 10.1109/TIP.2003.819861]
Xiong W, Yu J H, Lin Z, Yang J M, Lu X, Barnes C and Luo J B. 2019. Foreground-aware image inpainting//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 5840-5848[DOI: 10.1109/CVPR.2019.00599http://dx.doi.org/10.1109/CVPR.2019.00599]
Yeh R A, Chen C, Lim T Y, Schwing A G, Hasegawa-Johnson M and Do M N. 2017. Semantic image inpainting with deep generative models//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 6882-6890[DOI: 10.1109/CVPR.2017.728http://dx.doi.org/10.1109/CVPR.2017.728]
Yoo D, Kim N, Park S, PaekIn A S and Kweon I S. 2016. Pixel-level domain transfer//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 517-532[DOI: 10.1007/978-3-319-46484-8_31http://dx.doi.org/10.1007/978-3-319-46484-8_31]
Yu J H, Lin Z, Yang J M, Shen X H, Lu X and Huang T S. 2018. Generative image inpainting with contextual attention//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 5505-5514[DOI: 10.1109/CVPR.2018.00577http://dx.doi.org/10.1109/CVPR.2018.00577]
Zhang R, Isola P and Efros A A. 2016. Colorful image colorization//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 649-666[DOI: 10.1007/978-3-319-46487-9_40http://dx.doi.org/10.1007/978-3-319-46487-9_40]
相关文章
相关作者
相关机构
京公网安备11010802024621