Current Issue Cover
基于卷积自编码生成式对抗网络的 高分辨率破损图像修复

侯向丹, 刘昊然, 刘洪普(河北工业大学人工智能与数据科学学院, 天津 300401)

摘 要
目的 破损图像修复是一项具有挑战性的任务,其目的是根据破损图像中已知内容对破损区域进行填充。许多基于深度学习的破损图像修复方法对大面积破损的图像修复效果欠佳,且对高分辨率破损图像修复的研究也较少。对此,本文提出基于卷积自编码生成式对抗网络(convolutional auto-encoder generative adversarial network,CAE-GAN)的修复方法。方法 通过训练生成器学习从高斯噪声到低维特征矩阵的映射关系,再将生成器生成的特征矩阵升维成高分辨率图像,搜索与待修复图像完好部分相似的生成图像,并将对应部分覆盖到破损图像上,实现高分辨率破损图像的修复。结果 通过将学习难度较大的映射关系进行拆分,降低了单个映射关系的学习难度,提升了模型训练效果,在4个数据集上对不同破损程度的512×512×3高分辨率破损图像进行修复,结果表明,本文方法成功预测了大面积缺失区域的信息。与CE(context-encoders)方法相比,本文方法在破损面积大的图像上的修复效果提升显著,峰值信噪比(peak signal to noise ratio,PSNR)和结构相似性(structural similarity,SSIM)值最高分别提升了31.6% 和18.0%,与DCGAN(deep convolutional generative adversarial network)方法相比,本文方法修复的图像内容符合度更高,破损区域修复结果更加清晰,PSNR和SSIM值最高分别提升了24.4% 和50.0%。结论 本文方法更适用于大面积破损图像与高分辨率图像的修复工作。
关键词
High-resolution damaged images restoration based on convolutional auto-encoder generative adversarial network

Hou Xiangdan, Liu Haoran, Liu Hongpu(School of Artificial Intelligence and Data Science, Hebei University of Technology, Tianjin 300401, China)

Abstract
Objective The integrity of information transmission can be achieved if the image is intact currently. However, the required image files are often damaged or obscured, such as the damage of old photos, and the obscuration of the required content in the surveillance image. The purpose of damaged images restoration is to fill the damaged part in terms of the recognized region in the damaged image. The regular method of image restoration inserts the damaged area in accordance with the surrounding information based on texture synthesis technology linearly. Although this type of method can repair the texture, it lacks the manipulation of the global structure and image semantics of the damaged image. Deep-learning-based damaged image restoration methods have been illustrated via the classical context-encoders model. Although this method can perform better restoration on the color and content of the damaged image, the effect on the detail texture restoration is not ideal, and the restoration result appears blurred. When the damaged area is large, the repair effect is not qualified due to the lack of available information. Simultaneously, there are fewer analyses on high-resolution damaged image restoration now. Most of the existing damaged image restoration experiments use 128×128×3 and smaller images, and there are fewer experiments to repair 512×512×3 and larger images. In order to solve the two problems of large-area damaged image repair and high-resolution image repair, this analysis demonstrates a restoration method based on convolutional auto-encoder generative adversarial network (CAE-GAN). Method The generator is trained to learn the mapping relationship from Gaussian noise to the low-dimensional feature matrix, and then the generated feature matrix is upgraded to a high-resolution image, and the generated image similar to the intact part of the image to be repaired is sorted out. The corresponding part restoration on the damaged image to complete the repair of the high-resolution damaged image. First, high-resolution images are encoded and then decoded via the convolutional auto-encoder training part. Then, the parameters fix of the convolutional auto-encoder is adopted to assist in training the adversarial generation network part. The generator can generate different codes based on random Gaussian noise and then be decoded into high-resolution images based on the trained decoder. At the end, an overall connected network training for search can generate appropriate noise. After the noise is up-sampled by the generator and decoder, it will output a generated image similar to the image to be repaired, and cover the corresponding part on the damaged image, and then realize the repair of high-resolution damaged images. Result By segmenting the mapping relationships that are difficult to learn, the learning barriers of a single mapping relationship is declined, and the model training effect is improved. The repair experiments are conducted on the CelebA dataset, the street view house number(SVHN) dataset, the Oxford 102 flowers dataset, and the Stanford cars dataset. This demonstration illustrates that the method predicts the information of a large area of missing areas in a good way. Compared with the context-encoders(CE) method, the method improves the restoration effect on images with large damaged areas significantly. The content of the repaired damaged area is closer to the related intact parts, and the texture connection is smoother. The peak signal to noise ratio (PSNR) value can be increased to 31.6%, and the structural similarity (SSIM) value can be increased to 18.0%. The PSNR value can be increased by 24.4%, and the SSIM value can be increased by 50.0%. Conclusion Therefore, the method is suitable for the image restoration based on large-area damaged images and high-resolution images.
Keywords

订阅号|日报