浏览全部资源
扫码关注微信
Published： 16 May 2022 ，
Accepted： 26 October 2021
移动端阅览
Jixin Liu, Rui Chen, Shipeng An. Reference prior and generative prior linked distorted old photos restoration. [J]. Journal of Image and Graphics 27(5):1657-1668(2022)
目的
2
修复老照片具有重要的实用价值，但老照片包含多种未知复杂的退化，传统的修复方法组合不同的数字图像处理技术进行修复，通常产生不连贯或不自然的修复结果。基于深度学习的修复方法虽然被提出，但大多集中于对单一或有限的退化进行修复。针对上述问题，本文提出一种融合参考先验与生成先验的生成对抗网络来修复老照片。
方法
2
对提取的老照片和参考图像的浅层特征进行编码获得深层语义特征与潜在编码，对获得的潜在编码进一步融合获得深度语义编码，深度语义编码通过生成先验网络获得生成先验特征，并且深度语义编码引导条件空间多特征变换条件注意力块进行参考语义特征、生成先验特征与待修复特征的空间融合变换，最后通过解码网络重建修复图像。
结果
2
实验与6种图像修复方法进行了定量与定性评估。比较了4种评估指标，本文算法的所有指标评估结果均优于其他算法，PSNR (peak signal-to-noise ratio)为23.69 dB，SSIM (structural similarity index)为0.828 3，FID (Fréchet inception distance)为71.53，LPIPS (learned perceptual image patch similarity)为0.309)，相比指标排名第2的算法，分别提高了0.75 dB
0.019 7
13.69%
19.86%。定性结果中，本文算法具有更好的复杂退化修复能力，修复的细节更加丰富。此外，本文算法相比对比算法更加轻量，推断速度更快，以43.44 M的参数量完成256×256像素分辨率图像推断仅需248 ms。
结论
2
本文提出了融合参考先验与生成先验的老照片修复方法，充分利用了参考先验的语义信息与生成模型封装的人像先验，在主观与客观上均取得了先进的修复性能。
Objective
2
Distorted old photos restoration is a challenging issue in practice. Photos are severely eroded in harsh environments
resulting in unclear photo content or even permanent damage
such as scratches
noise
blur and color fading. First
distorted old photos are digitized and implemented (such as Adobe Photoshop) to harness pixel-level manual fine restoration via image processing software. However
manual restoration is time consuming and a batch of manual restoration is more challenged. Traditional methods restore distorted photos (such as digital filtering
edge detection
image patching
etc.) based on multiple restoration algorithms. However
incoherent or unclear restoration results are produced. A large number of deep learning methods have been facilitated nowadays. However
most of the deep learning methods originated from single degradation or several integrated degradations affect generalization ability because the synthesized artificial data cannot represent the real degradation process and data distribution. Based on the framework of generative adversarial network
our problem solving restores distorted old photos through introducing reference priors and generative priors
which improve the restoration quality and generalization performance of distorted old photos.
Method
2
The reference image option is a key factor to implement our method. A high-quality reference image is linked to the following features: 1) Structure similarity: the reference image and the distorted old photos should be similar to image structure. 2) Feature similarity: the distorted old photos restoration focuses more on the restoration of portraits. The resolution of the previous camera was generally not high and portraits are the core of the photo. The portrait content of the reference image should be as similar as possible to the portrait content in the targeted photos
including gender
age
posture
etc. Theoretically
the closer the two images are
the better the similarity coupling between features
more effective prior information can be obtained. Our method picks potential reference images up based on 2 portrait datasets of CelebFaces Atributes Dataset(CelebA) and Flickr faces high quality(FFHQ)
using structural similarity as an indicator. The image structural similarity is greater than 0.9 as an appropriated reference image; the reference image is further aligned with the distorted old photo through feature point detection. Our demonstration first extracts the shallow features of the reference image and the distorted old photos. The method uses a 3×3 convolution to extract the reference image features and uses 3 kernel sizes (7×7
5×5
3×3) convolutions to extract the shallow features of targeted photos. The shallow features of the reference image and the targeted photos are then encoded each to obtain deep semantics features in multiple-scales and latent semantic codes. Our 2 latent semantic codes are fused in latent space to obtain deep semantic codes through a series of overall interlinked layers. Deep semantic codes use the generative prior via compressed pre-trained generative model to generate generative prior features and guide spatial multi-feature (SMF) transformation condition attention block to fuse reference semantic features
generative prior features and distorted old photo features. Specifically
the distorted photo features are segmented into two sections
one section remains identity connection to ensure the fidelity of the restoration
and its copy is fused with generative prior features simultaneously. The other one is projected to affine transformation via the compressed reference semantic features. Finally
the 2 sections are interconnected and then the deep semantic codes are used for attention fusion. The fused features are related to the decoded features through the skip connection and residual connection
a following 3×3 convolution is used to reconstruct the restored photos. We build up a distorted old photo dataset excluded synthetic data.
Result
2
Our quantitative illustrations compares the results of the method with 6 state-of-the-art methods on 4 evaluation metrics
including signal-to-noise ratio (PSNR)
the structural similarity index (SSIM)
the learned perceptual image patch similarity (LPIPS) and Fréchet inception distance (FID)
which comprehensively consider the average pixel error
structural similarity
data distribution
and so on. Our demonstration is significantly better than other comparison methods in all evaluation metrics. The analyzed results of all numerical metrics are illustrated as mentioned below: the PSNR is 23.69 dB
the SSIM is 0.828 3
the LPIPS is 0.309 and the FID is 71.53
which are improved by 0.75 dB
0.019 7
13.69%
and 19.86%
respectively. Our qualitative method compares all the results of restoration methods. The best structured defects restoration quality is significantly better than other methods and the restoration results are more consistent and natural
such as missing
scratches
etc.
our unstructured defects method also facilitates comparable and better restoration results. Fewer parameters (43.44 M) and faster inference time are obtained (mean 248 ms for 256×256 resolution distorted old photos).
Conclusion
2
Our reference priors and generative priors' method restore distorted old photos. The semantic information of reference priors and generative model compressed portrait priors are facilitated to qualitative and quantitative restoration both.
深度学习生成对抗网络(GAN)老照片修复参考先验生成先验空间特征变换编解码网络多尺度感知
deep learninggenerative adversarial network(GAN)distorted old photos restorationreference priorgenerative priorspatial feature transformationencoder-decoder networkmulti-scale perception
Chen C F, Li X M, Zhou S C, Lin X H, Zuo W M and Zhang L. 2020. Blind face restoration via deep multi-scale component dictionaries//Proceedings of the 16th European Conference on Computer Vision. Glasgow, UK: Springer: 399-415[DOI: 10.1007/978-3-030-58545-7_23http://dx.doi.org/10.1007/978-3-030-58545-7_23]
Gu J J, Shen Y J and Zhou B L. 2020. Image processing using multi-code GAN prior//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 3009-3018[DOI: 10.1109/CVPR42600.2020.00308http://dx.doi.org/10.1109/CVPR42600.2020.00308]
Isola P, Zhu J Y, Zhou T H and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5967-5976[DOI: 10.1109/CVPR.2017.632http://dx.doi.org/10.1109/CVPR.2017.632]
Karras T, Laine S and Aila T. 2019. A style-based generator architecture for generative adversarial networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 4396-4405[DOI: 10.1109/CVPR.2019.00453http://dx.doi.org/10.1109/CVPR.2019.00453]
Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J and Aila T. 2020. Analyzing and improving the image quality of StyleGAN//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 8107-8116[DOI: 10.1109/CVPR42600.2020.00813http://dx.doi.org/10.1109/CVPR42600.2020.00813]
Lempitsky V, Vedaldi A and Ulyanov D. 2018. Deep image prior//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 9446-9454[DOI: 10.1109/CVPR.2018.00984http://dx.doi.org/10.1109/CVPR.2018.00984]
Li X M, Li W Y, Ren D W, Zhang H Z, Wang M and Zuo W M. 2020. Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 2703-2712[DOI: 10.1109/CVPR42600.2020.00278http://dx.doi.org/10.1109/CVPR42600.2020.00278]
MenonS, Damian A, Hu S J, Ravi N and Rudin C. 2020. PULSE: self-supervised photo upsampling via latent space exploration of generative models//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 2434-2442[DOI: 10.1109/CVPR42600.2020.00251http://dx.doi.org/10.1109/CVPR42600.2020.00251]
Nazeri K, Ng E, Joseph T, Qureshi F Z and Ebrahimi M. 2019. EdgeConnect: generative image inpainting with adversarial edge learning[EB/OL]. [2021-06-19].https://arxiv.org/pdf/1901.00212.pdfhttps://arxiv.org/pdf/1901.00212.pdf
Sagong M C, Shin Y G, Kim S W, Park S and Ko S J. 2019. PEPSI: fast image inpainting with parallel decoding network//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 11352-11360[DOI: 10.1109/CVPR.2019.01162http://dx.doi.org/10.1109/CVPR.2019.01162]
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2021-06-19].https://arxiv.org/pdf/1409.1556.pdfhttps://arxiv.org/pdf/1409.1556.pdf
Suganuma M, Liu X and Okatani T. 2019. Attention-based adaptive selection of operations for image restoration in the presence of unknown combined distortions//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 9031-9040[DOI: 10.1109/CVPR.2019.00925http://dx.doi.org/10.1109/CVPR.2019.00925]
Wan Z Y, Zhang B, Chen D D, Zhang P, Chen D, Liao J and Wen F. 2020. Bringing old photos back to life//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 2744-2754[DOI: 10.1109/CVPR42600.2020.00282http://dx.doi.org/10.1109/CVPR42600.2020.00282]
Wei K X, Fu Y, Yang J L and Huang H. 2020. A physics-based noise formation model for extreme low-light raw denoising//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 2755-2764[DOI: 10.1109/CVPR42600.2020.00283http://dx.doi.org/10.1109/CVPR42600.2020.00283]
Yasarla R, Perazzi F and Patel V M. 2020. Deblurring face images using uncertainty guided multi-stream semantic networks. IEEE Transactions on Image Processing, 29: 6251-6263[DOI: 10.1109/TIP.2020.2990354]
Zamir S W, Arora A, Khan S, Hayat M, Khan F S, Yang M H and Shao L. 2021. Multi-stage progressive image restoration[EB/OL]. [2021-05-16].https://arxiv.org/pdf/2102.02808v2.pdfhttps://arxiv.org/pdf/2102.02808v2.pdf
Zhao L, Mo Q H, Lin S H, Wang Z Z, Zuo Z W, Chen H B, Xing W and Lu D M. 2020. UCTGAN: diverse image inpainting based on unsupervised cross-space translation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 5740-5749[DOI: 10.1109/CVPR42600.2020.00578http://dx.doi.org/10.1109/CVPR42600.2020.00578]
Zhou Y Q, Barnes C, Shechtman E and Amirghodsi S. 2021. TransFill: reference-guided image inpainting by merging multiple color and spatial transformations[EB/OL]. [2021-05-29].https://arxiv.org/pdf/2103.15982.pdfhttps://arxiv.org/pdf/2103.15982.pdf
相关文章
相关作者
相关机构
京公网安备11010802024621