浏览全部资源
扫码关注微信
纸质出版日期： 2021-03-16 ，
录用日期： 2020-07-06
移动端阅览
程德强, 郭昕, 陈亮亮, 寇旗旗, 赵凯, 高蕊. 多通道递归残差网络的图像超分辨率重建[J]. 中国图象图形学报, 2021,26(3):605-618.
Deqiang Cheng, Xin Guo, Liangliang Chen, Qiqi Kou, Kai Zhao, Rui Gao. Image super-resolution reconstruction from multi-channel recursive residual network[J]. Journal of Image and Graphics, 2021,26(3):605-618.
目的
2
基于神经网络的图像超分辨率重建技术主要是通过单一网络非线性映射学习得到高低分辨率之间特征信息关系来进行重建，在此过程中较浅网络的图像特征信息很容易丢失，加深网络深度又会增加网络训练时间和训练难度。针对此过程出现的训练时间长、重建结果细节信息较模糊等问题，提出一种多通道递归残差学习机制，以提高网络训练效率和图像重建质量。
方法
2
设计一种多通道递归残差网络模型，该模型首先利用递归方法将残差网络块进行复用，形成32层递归网络，来减少网络参数、增加网络深度，以加速网络收敛并获取更丰富的特征信息。然后采集不同卷积核下的特征信息，输入到各通道对应的递归残差网络后再一起输入到共用的重建网络中，提高对细节信息的重建能力。最后引入一种交叉学习机制，将通道1、2、3两两排列组合交叉相连，进一步加速不同通道特征信息融合、促进参数传递、提高网络重建性能。
结果
2
本文模型使用DIV2K（DIVerse 2K）数据集进行训练，在Set5、Set14、BSD100和Urban100数据集上进行测试，并与Bicubic、SRCNN（super-resolution convolutional neural network）、VDSR（super-resolution using very deep convolutional network）、LapSRN（deep Laplacian pyramid networks for fast and accurate super-resolution）和EDSR_baseline（enhanced deep residual networks for single image super-resolution_baseline）等方法的实验结果进行对比，结果显示前者获取细节特征信息能力提高，图像有了更清晰丰富的细节信息；客观数据方面，本文算法的数据有明显的提升，尤其在细节信息较多的Urban100数据集中PSNR（peak signal-to-noise ratio）平均分别提升了3.87 dB、1.93 dB、1.00 dB、1.12 dB和0.48 dB，网络训练效率相较非递归残差网络提升30%。
结论
2
本文模型可获得更好的视觉效果和客观质量评价，而且相较非递归残差网络训练过程耗时更短，可用于复杂场景下图像的超分辨率重建。
Objective
2
The limitations of external environment
hardware conditions
and network resources will cause the images we obtain in daily life to be low-resolution images
which will affect the accuracy of images used in other applications. Therefore
super-resolution reconstruction technology has become a very important research topic. This technique can be used to recover super-resolution images. High-resolution images can be reconstructed from the information relationship between high-resolution and low-resolution images. Obtaining the correspondence between high-resolution and low-resolution images is the key to image super-resolution reconstruction technology. It is a basic method for neural networks to solve the problem of image super-resolution by using the single-channel network to learn the feature information relationship between high resolution and low resolution. However
the feature information of the image is easily lost in the shallow layer
and the low utilization of the feature information leads to an unsatisfactory reconstruction effect when the image magnification is large
and the restoration ability of the image detail information is poor. Simply deepening the depth of the network will increase the training time and difficulty of the network
which will waste a large amount of hardware resources and time. A multi-channel recursive residual network model is proposed to solve these problems. This model can improve network training efficiency by iterating the residual network blocks and enhance the detailed information reconstruction capability through multi-channel and cross-learning mechanisms.
Method
2
A multi-channel recursive cross-residual network model is designed. The use of a large number of convolutional layers in the model explains why training takes a large amount of time. Fewer convolutional layers will reduce network reconstruction performance. Therefore
the method of recursive residual network blocks is used to deepen the network depth and speed up the network training. First
a multi-channel recursive cross-residual network model is designed. The model uses recursive multiplexing of residual network blocks to form a 32-layer recursive network
thereby reducing network parameters and increasing network depth. This model can speed up network training and obtain richer information. Then
the amount of feature information which has a great influence on reconstruction performance
obtained by deepening the network
is limited. Characteristic information is easily lost in the network. Therefore
multi-channel networks are used to obtain richer feature information
increase the access to information
and reduce the rate of information loss. This method can improve the ability of the network to reconstruct image detail information. Finally
the degree of information fusion in the network is increased to facilitate image super-resolution reconstruction. A multi-channel network cross-learning mechanism is introduced to speed up the fusion of feature information of different channels
promote parameter transfer
and effectively improve the training efficiency and information fusion degree.
Result
2
Experimental results measure the performance of the algorithm by using peak signal-to-noise ratio (PSNR)
structural similarity (SSIM)
and network training time. Bicubic
A+
super-resolution convolutional neural network(SRCNN)
super-resolution using very deep convolutional network(VDSR)
deep Laplacian pyramid networks for fast and accurate super-resolution(LapSRN)
and enhanced deep residual network for single image super-resolution-baseline(EDSR_baseline) are used for comparison in open datasets. Training is performed on the DIV2K(DIVerse 2K) dataset
where the network uses 800 as the training dataset and 100 as the validation dataset. Tests are then performed on the Set5
Set14
BSD100
and Urban100 datasets with 219 test data. Three reconstruction models are designed
which are enlarged at×2
×3
and×4 resolutions
to facilitate the comparison of common algorithms. In the experiments
experimental data and reconstructed images are analyzed in detail. Compared with traditional serial networks
recursive networks can improve network efficiency and reduce network computing time. Especially in the Urban100 data set with more details
the experiments show that compared with Bicubic
SRCNN
VDSR
LapSRN
and traditional series networks
average PSNR increases by 3.87 dB
1.93 dB
1.00 dB
1.12 dB
and 0.48 dB
respectively. The visual effect is also clearer than that of the previous algorithm. Compared with the traditional tandem network
network training efficiency is improved by 30%.
Conclusion
2
The proposed network overcomes the shortcomings of single-channel deep networks and accelerates network convergence and information fusion by adding recursive residual networks and cross-learning mechanisms. In addition
recursive residual networks can accelerate network convergence and solve problems such as gradients during network training. Experimental results show that compared with the existing reconstruction methods
this method can obtain higher PSNR and SSIM
and can improve substantially in images with more detailed information. Thus
this method has the advantages of short training time
low information redundancy
and better reconstruction effect. In the future
we will consider continuing to optimize the recursive network scale and network cross-learning mechanism.
超分辨重建多通道递归交叉残差网络模型
super-resolution reconstructionmulti-channelrecursioncrossresidual network model
Cao Y J, Jia L L, Chen Y X, Lin N and Li X X. 2018. Review of computer vision based on generative adversarial networks. Journal of Image and Graphics, 23(10): 1433-1449
曹仰杰, 贾丽丽, 陈永霞, 林楠, 李学相. 2018. 生成式对抗网络及其计算机视觉应用研究综述. 中国图象图形学报, 23(10): 1433-1449[DOI:10.11834/jig.180103]
Chang H, Yeung D Y and Xiong Y M. 2004. Super-resolution through neighbor embedding//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC, USA: IEEE: #1315043[DOI: 10.1109/CVPR.2004.1315043http://dx.doi.org/10.1109/CVPR.2004.1315043]
Chen L L, Kou Q Q, Cheng D Q and Yao J. 2020. Content-guided deep residual network for single image super-resolution. Optik, 202: #163678[DOI:10.1016/j.ijleo.2019.163678]
Cheng D Q, Chen L L, Cai Y C, You D L and Tu Y L. 2018a. Image super-resolution reconstruction based on multi-dictionary and edge fusion. Journal of China Coal Society, 43(7): 2084-2090
程德强, 陈亮亮, 蔡迎春, 游大磊, 屠屹磊. 2018a. 边缘融合的多字典超分辨率图像重建算法. 煤炭学报, 43(7): 2084-2090[DOI:10.13225/j.cnki.jccs.2017.1263]
Cheng D Q, Liu W L, Shao L R and Chen L L. 2018b. Super resolution reconstruction algorithm based on kernel sparse representation and atomic correlation. Journal of Image and Graphics, 23(9): 1285-1292
程德强, 刘威龙, 邵丽蓉, 陈亮亮. 2018b. 核稀疏表示和原子相关度的图像重建. 中国图象图形学报, 23(9): 1285-1292[DOI:10.11834/jig.180011]
Dong C, Loy C C, He K M and Tang X O. 2014. Learning a deep convolutional network for image super-resolution//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 184-199[DOI: 10.1007/978-3-319-10593-2_13http://dx.doi.org/10.1007/978-3-319-10593-2_13]
Dong C, Loy C C and Tang X O. 2016. Accelerating the super-resolution convolutional neural network//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 391-407[DOI: 10.1007/978-3-319-46475-6_25http://dx.doi.org/10.1007/978-3-319-46475-6_25]
Dou X Y, Li C Y, Shi Q and Liu M X. 2020. Super-resolution for hyperspectral remote sensing images based on the 3D attention-SRGAN network. Remote Sensing, 12(7): 1204[DOI:10.3390/rs12071204]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern recognition (CVPR), Las Vegas, USA: IEEE: 770-778[DOI: 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90]
Huang J B, Singh A and Ahuja N. 2015. Single image super-resolution from transformed self-exemplars//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE: 5197-5206[DOI: 10.1109/CVPR.2015.7299156http://dx.doi.org/10.1109/CVPR.2015.7299156]
Jiang K, Wang Z Y, Yi P, Chen C, Huang B J, Luo Y M, Ma J Y and Jiang J J. 2020. Multi-scale progressive fusion network for single image deraining[EB/OL].[2020-03-24].https://arxiv.org/pdf/2003.10985v2.pdfhttps://arxiv.org/pdf/2003.10985v2.pdf
Kim J, Lee J K and Lee K M. 2016. Accurate image super-resolution using very deep convolutional networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 1646-1654[DOI: 10.1109/CVPR.2016.182http://dx.doi.org/10.1109/CVPR.2016.182]
Lai W S, Huang J B, Ahuja N and Yang M H. 2017. Deep Laplacian pyramid networks for fast and accurate super-resolution//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 5838-5843[DOI: 10.1109/CVPR.2017.618http://dx.doi.org/10.1109/CVPR.2017.618]
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z H and Shi W Z. 2016. Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 105-114[DOI: 10.1109/CVPR.2017.19http://dx.doi.org/10.1109/CVPR.2017.19]
Li H Y, Li C G, An J B and Ren J L. 2019. Attention mechanism improves CNN remote sensing image object dedection. Journal of Image and Graphics, 24(8): 1400-1408
李红艳, 李春庚, 安居白, 任俊丽. 2019. 注意力机制改进卷积神经网络的遥感图像目标检测. 中国图象图形学报, 24(8): 1400-1408[DOI:10.11834/jig.180649]
Li X and Orchard M T. 2001. New edge-directed interpolation. IEEE Transactions on Image Processing, 10(10): 1521-1527[DOI:10.1109/83.951537]
Lim B, Son S, Kim H, Nah S and Lee K M. 2017. Enhanced deep residual networks for single image super-resolution//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern recognition workshops (CVPRW). Honolulu, USA: IEEE: 1132-1140[DOI: 10.1109/CVPRW.2017.151http://dx.doi.org/10.1109/CVPRW.2017.151]
Liu S D, Wang X M and Zhang Y. 2019. Symmetric residual convolution neural networks for the image super-resolution reconstruction. Journal of Xidian University, 46(5): 15-23
刘树东, 王晓敏, 张艳. 2019. 一种对称残差CNN的图像超分辨率重建方法. 西安电子科技大学学报, 46(5): 15-23[DOI:10.19665/j.issn1001-2400.2019.05.003]
Lyu F F, Lu F, Wu J H and Lim C. 2018. MBLLEN: low-light image/video enhancement using CNNs[EB/OL].[2020-03-24].http://www.bmva.org/bmvc/2018/contents/papers/0700.pdfhttp://www.bmva.org/bmvc/2018/contents/papers/0700.pdf
Nah S, Kim T H and Lee K M. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 257-265[DOI: 10.1109/CVPR.2017.35http://dx.doi.org/10.1109/CVPR.2017.35]
Peled S and Yeshurun Y. 2001. Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging. Magnetic Resonance in Medicine, 45(1): 29-35[DOI:10.1002/1522-2594(200101)45:1<29::aid-mrm1005>3.0.co;2-z]
Peleg S, Keren D and Schweitzer L. 1987. Improving image resolution using subpixel motion. Pattern Recognition Letters, 5(3): 223-226[DOI:10.1016/0167-8655(87)90067-5]
Schultz R R and Stevenson R L. 1996. Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing, 5(6): 996-1011[DOI:10.1109/83.503915]
Shi W Z, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D and Wang Z H. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 1874-1883[DOI: 10.1109/CVPR.2016.207http://dx.doi.org/10.1109/CVPR.2016.207]
Sun J, Xu Z B and Shum H Y. 2008. Image super-resolution using gradient profile prior//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, USA: IEEE: 1-8[DOI: 10.1109/CVPR.2008.4587659http://dx.doi.org/10.1109/CVPR.2008.4587659]
Tan K, Wang X and Du P J. 2019. Research progress of the remote sensing classification combining deep learning and semi-supervised learning. Journal of Image and Graphics, 24(11): 1823-1841
谭琨, 王雪, 杜培军. 2019. 结合深度学习和半监督学习的遥感影像分类进展. 中国图象图形学报, 24(11): 1823-1841[DOI:10.11834/jig.190348]
Timofte R, De Smet V and Van Gool L. 2014. A+: adjusted anchored neighborhood regression for fast super-resolution//Proceedings of the 12th Asian Conference on Computer Vision. Singapore, Republic of Singapore: Springer: 111-126[DOI: 10.1007/978-3-319-16817-3_8http://dx.doi.org/10.1007/978-3-319-16817-3_8]
Timofte R, De V and Gool L V. 2013. Anchored neighborhood regression for fast example-based super-resolution//Proceedings of 2013 IEEE International Conference on Computer Vision.Sydney, Australia: IEEE: 1920-1927[DOI: 10.1109/ICCV.2013.241http://dx.doi.org/10.1109/ICCV.2013.241]
Unser M, Aldroubi A and Eden M. 1991. Fast B-spline transforms for continuous image representation and interpolation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3): 277-285[DOI:10.1109/34.75515]
Xiao J S, Liu E Y, Zhu L and Lei J F. 2017. Improved image super-resolution algorithm based on convolutional neural network. Acta Optica Sinica, 37(3): 0318011
肖进胜, 刘恩雨, 朱力, 雷俊锋. 2017. 改进的基于卷积神经网络的图像超分辨率算法. 光学学报, 37(3): 0318011[DOI:10.3788/aos201737.0318011]
Yang J C, Wright J, Huang T and Ma Y. 2008. Image super-resolution as sparse representation of raw image patches//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, USA: IEEE: 1-8[DOI: 10.1109/CVPR.2008.4587647http://dx.doi.org/10.1109/CVPR.2008.4587647]
Zhang W, Qu C F, Ma L, Guan J W and Huang R. 2016. Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network. Pattern Recognition, 59: 176-187[DOI:10.1016/j.patcog.2016.01.034]
相关作者
相关机构
京公网安备11010802024621