Single-image super-resolution construction based on multi-scale dense residual network

Zilu Ying; Xiang Long

doi:10.11834/jig.180431

Image Understanding and Computer Vision | Views : 0 下载量: 4 CSCD: 11

PDF
Export
Share
Collection
Album

Single-image super-resolution construction based on multi-scale dense residual network
Vol. 24, Issue 3, Pages: 410-419(2019)
Received：05 July 2018，

Revised：2018-9-4，

Published：16 March 2019
DOI： 10.11834/jig.180431
稿件说明：

移动端阅览

DOI：

Zilu Ying, Xiang Long. Single-image super-resolution construction based on multi-scale dense residual network[J]. Journal of Image and Graphics, 2019, 24(3): 410-419. DOI： 10.11834/jig.180431.

摘要

目的

近几年应用在单幅图像超分辨率重建上的深度学习算法都是使用单种尺度的卷积核提取低分辨率图像的特征信息，这样很容易造成细节信息的遗漏。另外，为了获得更好的图像超分辨率重建效果，网络模型也不断被加深，伴随而来的梯度消失问题会使得训练时间延长，难度加大。针对当前存在的超分辨率重建中的问题，本文结合GoogleNet思想、残差网络思想和密集型卷积网络思想，提出一种多尺度密集残差网络模型。

方法

本文使用3种不同尺度卷积核对输入的低分辨率图像进行卷积处理，采集不同卷积核下的底层特征，这样可以较多地提取低分辨率图像中的细节信息，有利于图像恢复。再将采集的特征信息输入残差块中，每个残差块都包含了多个由卷积层和激活层构成的特征提取单元。另外，每个特征提取单元的输出都会通过短路径连接到下一个特征提取单元。短路径连接可以有效地缓解梯度消失现象，加强特征传播，促进特征再利用。接下来，融合3种卷积核提取的特征信息，经过降维处理后与3×3像素的卷积核提取的特征信息相加形成全局残差学习。最后经过重建层，得到清晰的高分辨率图像。整个训练过程中，一幅输入的低分辨率图像对应着一幅高分辨率图像标签，这种端到端的学习方法使得训练更加迅速。

结果

本文使用两个客观评价标准PSNR（peak signal-to-noise ratio）和SSIM（structural similarity index）对实验的效果图进行测试，并与其他主流的方法进行对比。最终的结果显示，本文算法在Set5等多个测试数据集中的表现相比于插值法和SRCNN算法，在放大3倍时效果提升约3.4 dB和1.1 dB，在放大4倍时提升约3.5 dB和1.4 dB。

结论

实验数据以及效果图证明本文算法能够较好地恢复低分辨率图像的边缘和纹理信息。

Abstract

Objective

Single-image super-resolution aims to generate a visually pleasing high-resolution image from its degraded low-resolution measurement. Single-image super-resolution is used in various computer vision tasks

such as security and surveillance imaging

medical imaging

and image generation. However

image super-resolution is an ill-posed inverse problem because of a multitude of solutions for any low-resolution input. In recent years

a series of convolution neural networks model has been proposed for single-image super-resolution. The deep learning algorithms applied to single-image super-resolution reconstructions have used single-scale convolutional kernels to extract feature information of low-resolution images

thereby causing omissions of detailed information easily. Moreover

to obtain better image super-resolution reconstruction effects

the network model is constantly deepened

and the accompanying problems of gradients vanished

thereby resulting in longer training time and difficulty. A multi-scale dense residual network model based on GoogleNet

residual network

and intensive convolution network ideas is proposed to address these existing super-resolution reconstruction problems.

Method

Different from the traditional single-scale feature extraction convolutional kernel

this study uses three different scales of convolution kernels 3×3

5×5

and 7×7 to perform convolution processing on the input low-resolution images and collects the underlying features of different convolution kernels. Therefore

more detailed information on low-resolution images

which are beneficial to image restoration

are extracted. Then

the collected feature information is inputted into the residual block. Each residual block contains a number of feature extraction units consisting of convolutional and active layers. In addition

the output of each feature extraction unit is connected to the next feature extraction unit through a short path. Short-path connections can effectively alleviate the disappearance of gradients

enhance the propagation of features

and promote the reuse of features. Then

the feature information extracted by the three convolution kernels is merged

and the feature information extracted by the 3×3 pixels convolution kernel is added after dimensionality reduction processing to form a global residual learning. After a final reconstruction

a clear

high-resolution image is obtained. Throughout the training process

an input low-resolution image corresponds to a high-resolution image tag. This end-to-end learning method results in faster training. We use the mean squared error as the loss function. The loss is minimized using stochastic gradient descent with the standard backpropagation. We use a training data of 1 000 images from DIV2k

and the flipped and rotated versions of the training images are considered. We rotate the original images by 90° and 270°.

Results

This study uses two objective evaluation criteria

namely

peak signal-to-noise ratio and structural similarity index to test the effect map of the experiment and compare it with other mainstream methods. The final results show that compared with interpolation method and SRCNN algorithm on Set5 dataset

the proposed algorithm improves approximately 3.4 dB and 1.1 dB at three times of magnification and 3.5 dB and 1.4 dB at four times of magnification

respectively.

Conclusion

We propose a multi-scale dense residual network for single-image super-resolution. The experimental data and the effect graph confirm that the proposed algorithm can better recover the edge and texture information of low-resolution images. However

our networks have a large number of parameters because our algorithm uses three channels to recover image details. Therefore

our algorithm requires more convergence time. We will also reduce the number of weight parameters by decomposing the convolution kernel.

关键词

Keywords

references

Zhang L, Wu X L. An edge-guided image interpolation algorithm via directional filtering and data fusion[J]. IEEE Transactions on Image Processing, 2006, 15(8):2226-2238.[DOI:10.1109/TIP.2006.877407]

Zhang K B, Gao X B, Tao D C, et al. Single image super-resolution with non-local means and steering kernel regression[J]. IEEE Transactions on Image Processing, 2012, 21(11):4544-4556.[DOI:10.1109/TIP.2012.2208977]

Timofte R, De V, Van Gool L. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, NSW, Australia: IEEE, 2013: 1920-1927.[ DOI: 10.1109/ICCV.2013.241 http://dx.doi.org/10.1109/ICCV.2013.241 ]

Timofte R, De Smet V, Van Gool L. A+: adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of 12th Asian Conference on Computer Vision. Singapore: Springer, 2014: 111-126.[ DOI: 10.1007/978-3-319-16817-3_8 http://dx.doi.org/10.1007/978-3-319-16817-3_8 ]

Peleg T, Elad M. A statistical prediction model based on sparse representations for single image super-resolution[J]. IEEE Transactions on Image Processing, 2014, 23(6):2569-2582.[DOI:10.1109/TIP.2014.2305844]

Hinton G E, Salakhutdinov R R.Reducing the Dimensionality of Data with Neural Networks[J].ScienceNew Series, 2006, 313(5786):504-507.[DOI:10.1126/science.1127647]

Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.[DOI:10.1109/TPAMI.2015.2439281]

Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 1874-1883.[ DOI: 10.1109/CVPR.2016.207 http://dx.doi.org/10.1109/CVPR.2016.207 ]

Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 1646-1654.[ DOI: 10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ]

He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 770-778.[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 1637-1645.[ DOI: 10.1109/CVPR.2016.181 http://dx.doi.org/10.1109/CVPR.2016.181 ]

Tai Y, Yang J, Liu X M. Image super-resolution via deep recursive residual network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii, USA: IEEE Computer Society, 2017: 2790-2798.[ DOI: 10.1109/CVPR.2017.298 http://dx.doi.org/10.1109/CVPR.2017.298 ]

Dong C, Loy C C, Tang X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer, 2016: 391-407.[ DOI: 10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25 ]

Tong T, Li G, Liu X J, et al. Image super-resolution using dense skip connections[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE Computer Society, 2018: 4809-4817.[ DOI: 10.1109/ICCV.2017.514 http://dx.doi.org/10.1109/ICCV.2017.514 ]

Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, HI, USA: IEEE, 2017: 1132-1140.

Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE, 2015: 1-9.

Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii, USA: IEEE Computer Society, 2017: 2261-2269.[ DOI: 10.1109/CVPR.2017.243 http://dx.doi.org/10.1109/CVPR.2017.243 ]

Lai W S, Huang J B, Ahuja N, et al. Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii, USA: IEEE Computer Society, 2017: 5835-5843.[ DOI: 10.1109/CVPR.2017.618 http://dx.doi.org/10.1109/CVPR.2017.618 ]

Alert me when the article has been cited

提交

Image classification network with random dilated convolution

Double-pooling residual classification network based on feature reordering attention mechanism

Chemical structure recognition method based on attention mechanism and encoder-decoder architecture

RTDNet： red tide detection network for high-resolution satellite images

A micro-hyperspectral image classification method of gallbladder cancer based on multi-scale fusion attention mechanism