Image super-resolution reconstruction via deep network based on edge-enhancement

Zhenzhu Xie; Congzhong Wu; Shu Zhan

doi:10.11834/jig.170312

Column of CACIS 2017 | Views : 0 下载量: 4 CSCD: 3

PDF
Export
Share
Collection
Album

Image super-resolution reconstruction via deep network based on edge-enhancement
Vol. 23, Issue 1, Pages: 114-122(2018)
Received：27 June 2017，

Revised：2017-8-23，

Published：16 January 2018
DOI： 10.11834/jig.170312
稿件说明：

移动端阅览

DOI：

Zhenzhu Xie, Congzhong Wu, Shu Zhan. Image super-resolution reconstruction via deep network based on edge-enhancement[J]. Journal of Image and Graphics, 2018, 23(1): 114-122. DOI： 10.11834/jig.170312.

摘要

目的

针对基于学习的图像超分辨率重建算法中存在边缘信息丢失、易产生视觉伪影等问题，提出一种基于边缘增强的深层网络模型用于图像的超分辨率重建。

方法

本文算法首先利用预处理网络提取输入低分辨率图像的低级特征，然后将其分别输入到两路网络，其中一路网络通过卷积层级联的卷积网络得到高级特征，另一路网络通过卷积网络和与卷积网络成镜像结构的反卷积网络的级联实现图像边缘的重建。最后，利用支路连接将两路网络的结果进行融合，并将其结果通过一个卷积层从而得到最终重建的具有边缘增强效果的高分辨率图像。

结果

以峰值信噪比（PSNR）和结构相似度（SSIM）作为评价指标来评价算法性能，在Set5、Set14和B100等常用测试集上放大3倍情况下进行实验，并且PSNR/SSIM指标分别取得了33.24 dB/0.9156、30.60 dB/0.852 1和28.45 dB/0.787 3的结果，相比其他方法有很大提升。

结论

定量与定性的实验结果表明，基于边缘增强的深层网络的图像超分辨重建算法所重建的高分辨率图像不仅在重建图像边缘信息方面有较好的改善，同时也在客观评价和主观视觉上都有很大提高。

Abstract

Objective

Image super-resolution reconstruction is a branch of image restoration

which concerns with the problem of generating a plausible and visually pleasing high-resolution output image from a low-resolution input image. This approach has many practical applications

ranging from video surveillance imaging to medical imaging and satellite remote-sensing image processing. Although some methods have achieved reasonable results in recent years

they have mainly focused on visual artifacts

while the loss of edge information has been rarely mentioned. To address these weaknesses

a novel image super-resolution reconstruction method via deep network based on edge enhancement is proposed in this study.

Method

Given that deep learning has demonstrated excellent performance in computer vision problems

some scholars have utilized convolutional neural networks to design deep architecture for image super resolution. Dong et al successfully introduced deep learning into a super-resolution-based method; they demonstrated that convolutional neural networks could be used to learn mapping from a low-resolution image to a high-resolution image in an end-to-end way and achieved state-of-the-art results. Besides

Inspired by semantic segmentation based on deconvolution network

we introduce a deconvolution network to reconstruct edge information. The proposed model considers an interpolated low-resolution image (to the desired size) as input. The preprocessed network is utilized to extract low-level features of the input image

which are imported into the mixture network. The mixture network consists of two roads. One road is used to obtain high-level features by cascading the convolutional layer many times

and the other road realizes the reconstruction of the image edge by cascading between the convolutional network and its mirror network-deconvolution network. The convolutional and deconvolution layers in stacked style can retain the feature map size by adding pad wise-pixel. We can obtain the final reconstruction result through a convolutional layer by fusing the two road results via bypass connection. We select the rectified linear unit as activation function in our model to accelerate the training process and avoid the vanishing gradient. We employ 91 images as the training set and observe their performance changes in Set5

Set14

and B100 with scaling factors of 2

and 4 respectively. The training set is further augmented by rotating the original image by 90°

180°

and 270° and flipping them upside down to prevent overfitting in deep network. Notably

we initially convert the color images of RGB space into YCbCr space

considering that human vision is more sensitive to details in intensity than in color. We then apply the proposed algorithm to the luminance Y channel

and the Cb

Cr channels are upscaled by bicubic interpolation.

Result

All experiments are implemented on the Caffe package. The proposed algorithm considers peak-signal-to-noise ratio and structural similarity index as evaluation metrics. The experimental results on Set5 for the scale factor of 3 are 33.24 dB/0.915 6

30.60 dB/0.852 1

and 27.99 dB/0.784 8. Compared with bicubic

ScSR

A+

SelfEx

SRCNN

and CSCN

the proposed algorithm shows improved performances by 2.85 dB/4.74

1.9 dB/2.87

0.66 dB/0.68

0.66 dB/0.63

0.49 dB/0.66

and 0.14 dB/0.12 respectively. The running time of GPU version on Set5 for scale factor of 3 only takes 0.62 s

which is obviously superior to those of the other methods.

Conclusion

Convolutional neural networks have been increasingly popular in image super-resolution reconstruction. This study employs a deep network that contains convolution

deconvolution

and unpooling

which is used for reconstructing image edge information. The experimental results demonstrate that the proposed method based on edge enhancement model achieves better quantitative and qualitative reconstruction performances than those of the other methods.

关键词

Keywords

references

Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873.[DOI:10.1109/TIP.2010.2050625]

Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. Avignon, France:Springer-Verlag, 2010:711-730.[ DOI:10.1007/978-3-642-27413-8_47 http://dx.doi.org/10.1007/978-3-642-27413-8_47 ] http://www.springerlink.com/content/56276x8370377023/ .

Timofte R, De Smet V, Van Gool L. A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of the 12th Asian Conference on Computer Vision. Singapore:Springer, 2015, 9006:111-126.[ DOI:10.1007/978-3-319-16817-3_8 http://dx.doi.org/10.1007/978-3-319-16817-3_8 ] http://link.springer.com/chapter/10.1007/978-3-319-16817-3_8 .

Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015:1-9.[ DOI:10.1109/CVPR.2015.7298594 http://dx.doi.org/10.1109/CVPR.2015.7298594 ] http://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=7298594 .

Dong C, Chen C L, He K C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.[DOI:10.1109/TPAMI.2015.2439281]

Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA:IEEE, 2016:1646-1654.[ DOI:10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ] http://arxiv.org/abs/1511.04587 .

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Proceedings of 2015 International Conference on Learning Representations.

Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:1520-1528.[ DOI:10.1109/ICCV.2015.178 http://dx.doi.org/10.1109/ICCV.2015.178 ]

Zeiler M D, Taylor G W, Fergus R. Adaptive deconvolutional networks for mid and high level feature learning[C]//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011:2018-2025.[ DOI:10.1109/ICCV.2011.6126474 http://dx.doi.org/10.1109/ICCV.2011.6126474 ]

Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada:Curran Associates Inc., 2012:1097-1105.

Kolen J, Kremer S. Gradient flow in recurrent nets:the difficulty of learning LongTerm dependencies[M]Wiley-IEEE Press, 2001.[DOI:10.1109/9780470544037.ch14]

Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[C]//Proceedings of the 13th European Conference. Zurich, Switzerland:Springer, 2013, 8689:818-833.[ DOI:10.1007/978-3-319-10590-1_53 http://dx.doi.org/10.1007/978-3-319-10590-1_53 ]

LeCun Y, Boser B, Denker J S, et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(4):541-551.[DOI:10.1162/neco.1989.1.4.541]

Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks[C]//Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. Sardinia, Italy:PMLR, 2010, 9:249-256.

Jia Y Q, Shelhamer E, Donahue J, et al. Caffe:convolutional architecture for fast feature embedding[C]//Proceedings of the 22nd ACM International Conference on Multimedia. Orlando, Florida, USA:ACM, 2014:675-678.[ DOI:10.1145/2647868.2654889 http://dx.doi.org/10.1145/2647868.2654889 ]

Huang J B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:I EEE, 2015:5197-5206.[ DOI:10.1109/CVPR.2015.7299156 http://dx.doi.org/10.1109/CVPR.2015.7299156 ]

Wang Z W, Liu D, Yang J C, et al. Deep networks for image super-resolution with sparse prior[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Washington, DC, USA:IEEE, 2015:370-378.[ DOI:10.1109/ICCV.2015.50 http://dx.doi.org/10.1109/ICCV.2015.50 ]

He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA:IEEE, 2016:770-778.[ DOI:10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Alert me when the article has been cited

提交

Edge-enhanced ultra high definition video quality assessment

Critical review of human cardiac magnetic resonance image super resolution reconstruction based on deep learning method

Image super-resolution reconstruction via deep network based on multi-staged fusion

Learning transferrable attention for joint balanced domain adaptation

Application of faster R-CNN in image defect detection of industrial CT