浏览全部资源
扫码关注微信
Received：17 October 2017，
Revised：2018-1-26，
Published：16 July 2018
移动端阅览
目的
2
基于学习的图像超分辨率重建方法已成为近年来图像超分辨率重建研究的热点。针对基于卷积神经网络的图像超分辨率重建(SRCNN)方法网络层少、感受野小、泛化能力差等缺陷，提出了基于中间层监督卷积神经网络的图像超分辨率重建方法，以进一步提高图像重建的质量。
方法
2
设计了具有中间层监督的卷积神经网络结构，该网络共有16层卷积层，其中第7层为中间监督层；定义了监督层误差函数和重建误差函数，用于改善深层卷积神经网络梯度消失现象。训练网络时包括图像预处理、特征提取和图像重建3个步骤，采用不同尺度因子(2、3、4)模糊的低分辨率图像交叉训练网络，以适应对不同模糊程度的图像重建；使用卷积操作提取图像特征时将参数
$$pad$$
设置为1，提高了对图像和特征图的边缘信息利用；利用残差学习完成高分辨率图像重建。
结果
2
在Set5和Set14数据集上进行了实验，并和双三次插值、A+、SelfEx和SRCNN等方法的结果进行比较。在主观视觉评价方面，本文方法重建图像的清晰度和边缘锐度更好。客观评价方面，本文方法的峰值信噪比(PSNR)平均分别提高了2.26 dB、0.28 dB、0.28 dB和0.15 dB，使用训练好的网络模型重建图像耗用的时间不及SRCNN方法的一半。
结论
2
实验结果表明，本文方法获得了更好的主观视觉评价和客观量化评价，提升了图像超分辨率重建质量，泛化能力好，而且图像重建过程耗时更短，可用于自然场景图像的超分辨率重建。
Objective
2
Learning-based image super-resolution reconstruction has recently become a research hotspot. A new image super-resolution reconstruction method based on intermediate supervision convolutional neural network (CNN) is proposed to solve the problems of less network layers
small receptive field
and functionality that is limited to a single scale in the original network super-resolution CNN (SRCNN) to further improve the quality of image reconstruction.
Method
2
This method is based on the idea of deep learning CNN. First
when information regarding the input or gradient passes through many layers
such information can vanish and be "washed out" by the time it reaches the end (or beginning) of the network. Therefore
we design a CNN structure that has an intermediate supervision layer. The learning architecture has 16 weight layers
and information used for reconstruction (receptive field) is considerably substantial (31×31 versus 13×13). The learning architecture layers are the same type
except for the first
seventh
and the last layers:64 filters with a size of 3×3×64
where a filter operates on a 3×3 spatial region across 64 channels (feature maps). Each convolutional layer is followed by a rectified linear unit as an activation function. The first convolution layer operates on the input image. The seventh layer is an intermediate supervision layer that can guide the training of preceding layers in the CNN; this guidance can be considered an implicit deep supervision adopted to strengthen the learning capability during training. The last layer
which uses a single filter with size 3×3×64
is used for image reconstruction. Second
the supervision layer and reconstruction loss functions are defined to solve the vanishing gradient problem of the deep CNN. The training procedure includes three steps:image preprocessing
feature extraction
and image reconstruction. In the first step
the network is trained by the low-resolution images
which are blurred by different upscaling factors (2
3
4
possibly including fractional factors)
to reconstruct different degrees of blurred images well. In the second step
the image feature is extracted using convolution operations. Unlike center pixels in the SRCNN
those in the center-surround relation methodology are inferred by surrounding pixels that are not fully utilized. We pad one before the convolutions to keep the sizes of all feature maps (including the output image) uniform
thereby increasing the use of edge information for images and feature maps. In the last step
a smooth loss function with a good generalization performance is easily achieved with a comprehensive use of the features of shallow complexity because the input and output (predicted) images have high similarity and the high-resolution image is reconstructed by the residual learning method.
Result
2
The proposed method is evaluated on open challenge datasets Set5 and Set14
which are often used in super-resolution methods. Experimental results show that the proposed method has better subjective visual effect and objective quantitative evaluation than bicubic interpolation
A+
SelfEx
and SRCNN. For subjective visual evaluation
the proposed method produces a reconstructed image that has superior clarity and edge sharpness. For objective evaluation
the average peak signal to noise ratio (PSNR) achieved by this method is 2.26 dB
0.28 dB
0.28 dB
and 0.15 dB higher
respectively
than those attained by the other approaches. Meanwhile
the time consumed is less than half that of the SRCNN method when using the trained network models to reconstruct images.
Conclusion
2
The flow of information and gradients can be smoothly propagated throughout the entire network by introducing intermediate supervision into our network
thereby enhancing the reconstruction capability of networks and the training efficiency. Extensive experiments confirm that the proposed method
which has intermediate supervision
improves the quality and efficiency of image super-resolution reconstruction. This approach has good generalization capability and can be used for the super-resolution reconstruction of natural scene images.
Zhang Y, Li J Z, Li D L, et al. Super-resolution reconstruction for UAV video[J]. Journal of Image and Graphics, 2016, 21(7):967-976.
张岩, 李建增, 李德良, 等.无人机侦察视频超分辨率重建方法[J].中国图象图形学报, 2016, 21(7):967-976. [DOI:10.11834/jig.20160715]
Zhao J J, Fang Q, Liang Z C, et al. Sketch face recognition based on super-resolution reconstruction[J]. Journal of Image and Graphics, 2016, 21(2):218-224.
赵京晶, 方琪, 梁植程, 等.超分辨率重建的素描人脸识别[J].中国图象图形学报, 2016, 21(2):218-224. [DOI:10.11834/jig. 20160211]
Hu C S, Zhan S, Wu C Z. Image super-resolution based on deep learning features[J]. Acta Automatica Sinica, 2017, 43(5):814-821.
胡长胜, 詹曙, 吴从中.基于深度特征学习的图像超分辨率重建[J].自动化学报, 2017, 43(5):814-821. [DOI:10.16383/j.aas.2017.c150634]
Su H, Zhou J, Zhang Z H. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8):1202-1213.
苏衡, 周杰, 张志浩.超分辨率图像重建方法综述[J].自动化学报, 2013, 39(8):1202-1213. [DOI:10.3724/SP.J.1004.2013.01202]
Chavez-Roman H, Ponomaryov V. Super resolution image generation using wavelet domain interpolation with edge extraction via a sparse representation[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(10):1777-1781.[DOI:10.1109/LGRS.2014.2308905]
Tai Y W, Liu S C, Brown M S, et al. Super resolution using edge prior and single image detail synthesis[C]//Proceedings of 2010 IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, USA: IEEE, 2010: 2400-2407. [ DOI:10.1109/CVPR.2010.5539933 http://dx.doi.org/10.1109/CVPR.2010.5539933 ]
Zhang K B, Gao X B, Tao D C, et al. Single image super-resolution with non-local means and steering kernel regression[J]. IEEE Transactions on Image Processing, 2012, 21(11):4544-4556.[DOI:10.1109/TIP.2012.2208977]
Xu J, Chang Z G, Fan J L. Image superresolution by midfrequency sparse representation and total variation regularization[J]. Journal of Electronic Imaging, 2015, 24(1):#013039.[DOI:10.1117/1.JEI.24.1.013039]
Zhang K B, Tao D C, Gao X B, et al. Learning multiple linear mappings for efficient single image super-resolution[J]. IEEE Transactions on Image Processing, 2015, 24(3):846-861.[DOI:10.1109/TIP.2015.2389629]
Deng C Z, Tian W, Wang S Q, et al. Super-resolution reconstruction of approximate sparsity regularized infrared images[J]. Optics and Precision Engineering, 2014, 22(6):1648-1654.
邓承志, 田伟, 汪胜前, 等.近似稀疏正则化的红外图像超分辨率重建[J].光学精密工程, 2014, 22(6):1648-1654. [DOI:10.3788/OPE.20142206.1648]
Chang H, Yeung D Y, Xiong Y M. Super-resolution through neighbor embedding[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2004: Ⅰ. [ DOI:10.1109/CVPR.2004.1315043 http://dx.doi.org/10.1109/CVPR.2004.1315043 ]
Cao X,Chen X H, Pan R H. Fast image super-resolution algorithm based on sparse representation[J]. Computer Engineering, 2015, 41(6):211-215, 220.
曹翔, 陈秀宏, 潘荣华.基于稀疏表示的快速图像超分辨率算法[J].计算机工程, 2015, 41(6):211-215, 220. [DOI:10.3969/j.issn. 1000-3428.2015.06.038]
Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. Avignon, France: Springer, 2012: 711-730. [ DOI:10.1007/978-3-642-27413-8_47 http://dx.doi.org/10.1007/978-3-642-27413-8_47 ]
Timofte R, De V, Van Gool L. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE, 2013: 1920-1927. [ DOI:10.1109/ICCV.2013.241 http://dx.doi.org/10.1109/ICCV.2013.241 ]
Timofte R, De Smet V, Van Gool L. A+: adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of the 12th Asian Conference on Computer Vision. Singapore: Springer, 2014: 111-126. [ DOI:10.1007/978-3-319-16817-3_8 http://dx.doi.org/10.1007/978-3-319-16817-3_8 ]
Schulter S, Leistner C, Bischof H. Fast and accurate image upscaling with super-resolution forests[C] //Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA, 2015: 3791-3799. [ DOI:10.1109/CVPR.2015.7299003 http://dx.doi.org/10.1109/CVPR.2015.7299003 ]
Dong C, Loy C C, He K M, et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer, 2014: 184-199. [ DOI:10.1007/978-3-319-10593-2 http://dx.doi.org/10.1007/978-3-319-10593-2 ]
He K M, Zhang X Y, Ren S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9):1904-1916.[DOI:10.1109/TPAMI.2015.2389824]
Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 1646-1654. [ DOI:10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ]
Xiao J S, Liu E Y, Zhu L, et al. Improved image super-resolution algorithm based on convolutional neural network[J]. Acta Optica Sinica, 2017, 37(3):#0318011.
肖进胜, 刘恩雨, 朱力, 等.改进的基于卷积神经网络的图像超分辨率算法[J].光学学报, 2017, 37(3):#0318011. [DOI:10.3788/AOS201737.0318011]
Li S M, Lei G Q, Fan R. Depth map super-resolution reconstruction based on convolutional neural networks[J]. Acta Optica Sinica, 2017, 37(12):#1210002.
李素梅, 雷国庆, 范如.基于卷积神经网络的深度图超分辨率重建[J].光学学报, 2017, 37(12):#1210002. [DOI:10.3788/AOS201737.1210002]
Xu K S, Wang H L, Tang P J. Image captioning with deep LSTM based on sequential residual[C]//Proceedings of 2017 IEEE International Conference on Multimedia and Expo. Hong Kong, China: IEEE, 2017: 361-366. [ DOI:10.1109/ICME.2017.8019408 http://dx.doi.org/10.1109/ICME.2017.8019408 ]
Sun X, Li X G, Li J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5):697-709.
孙旭, 李晓光, 李嘉锋, 等.基于深度学习的图像超分辨率复原研究进展[J].自动化学报, 2017, 43(5):697-709. [DOI:10.16383/j.aas.2017.c 160629]
Wei S E, Ramakrishna V, Kanade T, et al. Convolutional pose machines[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 4724-4732. [ DOI:10.1109/CVPR.2016.511 http://dx.doi.org/10.1109/CVPR.2016.511 ]
He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778. [ DOI:10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Jia Y Q, Shelhamer E, Donahue J, et al. Caffe: convolutional architecture for fast feature embedding[C]//Proceedings of the 22nd ACM International Conference on Multimedia. Orlando, Florida: ACM, 2014: 675-678. [ DOI:10.1145/2647868.2654889 http://dx.doi.org/10.1145/2647868.2654889 ]
Sun Y W, Li L T, Cong P, et al. Super-resolution method for radiation image based on deep learning[J]. Atomic Energy Science and Technology, 2017, 51(5):890-895.
孙跃文, 李立涛, 丛鹏, 等.基于深度学习的辐射图像超分辨率重建方法[J].原子能科学技术, 2017, 51(5):890-895. [DOI:10.7538/yzk.2017.51.05.0890]
Huang J B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE, 2015: 5197-5206. [ DOI:10.1109/CVPR.2015.7299156 http://dx.doi.org/10.1109/CVPR.2015.7299156 ]
相关文章
相关作者
相关机构
京公网安备11010802024621
