浏览全部资源
扫码关注微信
收稿：2020-05-13，
修回：2020-10-19，
录用：2020-10-26，
纸质出版：2021-04-16
移动端阅览
目的
2
单幅图像超分辨率重建的深度学习算法中，大多数网络都采用了单一尺度的卷积核来提取特征（如3×3的卷积核），往往忽略了不同卷积核尺寸带来的不同大小感受域的问题，而不同大小的感受域会使网络注意到不同程度的特征，因此只采用单一尺度的卷积核会使网络忽略了不同特征图之间的宏观联系。针对上述问题，本文提出了多层次感知残差卷积网络（multi-level perception residual convolutional network，MLP-Net，用于单幅图像超分辨率重建）。
方法
2
通过特征提取模块提取图像低频特征作为输入。输入部分由密集连接的多个多层次感知模块组成，其中多层次感知模块分为浅层多层次特征提取和深层多层次特征提取，以确保网络既能注意到图像的低级特征，又能注意到高级特征，同时也能保证特征之间的宏观联系。
结果
2
实验结果采用客观评价的峰值信噪比（peak signal to noise ratio，PSNR）和结构相似性（structural similarity，SSIM）两个指标，将本文算法其他超分辨率算法进行了对比。最终结果表明本文算法在4个基准测试集上（Set5、Set14、Urban100和BSD100（Berkeley Segmentation Dataset））放大2倍的平均峰值信噪比分别为37.851 1 dB，33.933 8 dB，32.219 1 dB，32.148 9 dB，均高于其他几种算法的结果。
结论
2
本文提出的卷积网络采用多尺度卷积充分提取分层特征中的不同层次特征，同时利用低分辨率图像本身的结构信息完成重建，并取得不错的重建效果。
Objective
2
Single image super-resolution reconstruction (SISR) is a classic problem in computer vision. SISR aims to reconstruct one high-resolution image from single or many low-resolution (LR) images. Currently
image super-resolution (SR) technology is widely used in medical imaging
satellite remote sensing
video surveillance
and other fields. However
the SR problem is an essentially complex and morbid problem. To solve this problem
many SISR methods have been proposed
including interpolation-based methods and reconstruction-based methods. Due to large amplification factors
the repair performance will drop sharply
and the reconstructed results are very poor. With the rise of deep learning
deep convolutional neural networks have also been used to solve this problem. Researchers have proposed a series of models and made significant progress. With the gradual understanding of deep learning techniques
researchers have found that deep network brings better results than shallow network
and too deep network can cause gradient explosion or disappearance. In addition
the gradient explosion or disappearance can cause the model to be untrainable and thus unable to achieve the best results through training. In recent years
most networks based on deep learning for single-image SR reconstruction adopt single-scale convolution kernels. Generally
a 3×3 convolution kernel is used for feature extraction. Although single-scale convolution kernels can also extract a lot of detailed information
these algorithms usually ignore the problem of different receptive field sizes caused by different convolution kernel sizes. Receptive fields of different sizes will make the network pay attention to different features; therefore
only using a 3×3 convolution kernel will cause the network to ignore the macroscopic relation between different feature images. Considering these problems
this study proposes a multi-level perception network based on GoogLeNet
residual network
and dense convolutional network.
Method
2
First
the feature extraction module is used as the input
which can extract low-frequency image features. The feature extraction module consists of two 3×3 convolution layers
which is input to multiple densely connected multi-level perception modules. The multi-level perception module is composed of 3×3 and 5×5 convolution kernels. The 3×3 convolution kernel is responsible for extracting detailed feature information
and the 5×5 convolution kernel is responsible for extracting global feature information. Second
the multi-level perception module is divided into shallow multi-level feature extraction
deep multi-level feature extraction
and tandem compression unit. The shallow multi-level feature extraction is composed of 3×3 chain convolution and 5×5 chain convolution. The former is responsible for extracting fine local feature information in shallow features
whereas the latter is responsible for extracting global features in shallow features. The deep multi-level feature extraction is also composed of 3×3 chain convolution and 5×5 chain convolution. The former extracts fine local feature information in deep features
whereas the latter extracts global feature information in deep features. In the tandem compression unit
the global feature information in shallow features
the fine local feature information in deep features
the global information in deep features
and the initial input are concatenated together and then compressed into the same dimension as the input image. In this way
not only low-level and high-level features of the image can be ensured
but also the macro relationship between the features can be guaranteed. Finally
the reconstruction module is used to obtain the final output by combining the upscaling image with the residual image. This study adopts the DIV2K dataset
which consists of 800 high-definition images
and each image has probably 2 million pixels. In order to make full use of these data
the picture is randomly rotated by 90°
180°
and 270° and horizontally flipped.
Result
2
The reconstructed results are evaluated by using the peak signal-to-noise ratio (PSNR) and structural similarity index and compared with some state-of-the-art SR reconstruction methods. The reconstructed results with 2 scaling factor show that the PSNRs of the proposed algorithm in four benchmark test sets (Set5
Set14
Berkeley Segmentation Dataset(BSD100)
and Urban100) are 37.851 1 dB
33.933 8 dB
32.219 1 dB
and 32.148 9 dB
respectively
which are all higher than those of the other methods.
Conclusion
2
Compared with other algorithms
the proposed convolutional network model in this study can better take into account the problem of the receptive field and fully extracts different levels of hierarchical features through multi-scale convolution. At the same time
the model uses the structural feature information of the LR image itself to complete the reconstruction
and good reconstructed results can be obtained by using this model.
Ahn N, Kang B and Sohn K A. 2018. Fast, accurate, and lightweight super-resolution with cascading residual network//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 256-272[ DOI: 10.1007/978-3-030-01249-6_16 http://dx.doi.org/10.1007/978-3-030-01249-6_16 ]
Dai T, Cai J R, Zhang Y B, Xia S Tand Zhang L. 2019. Second-order attention network for single image super-resolution//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 11057-11066[ DOI: 10.1109/CVPR.2019.01132 http://dx.doi.org/10.1109/CVPR.2019.01132 ]
Dong C, Loy C C, He K M and Tang X O. 2016a. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307[DOI:10.1109/TPAMI.2015.2439281]
Dong C, Loy C C and Tang X O. 2016b. Accelerating the super-resolution convolutional neural network//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 391-407[ DOI: 10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25 ]
Fang F M, Li J C and Zeng T Y. 2020. Soft-edge assisted network for single image super-resolution. IEEE Transactions on Image Processing, 29: 4656-4668[DOI:10.1109/TIP.2020.2973769]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Hu X C, Mu H Y, Zhang X Y, Wang Z L, Tan T N and Sun J. 2019. Meta-SR: a magnification-arbitrary network for super-resolution//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 1575-1584[ DOI: 10.1109/CVPR.2019.00167 http://dx.doi.org/10.1109/CVPR.2019.00167 ]
Huang G, Liu Z, Van Der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2261-2269[ DOI: 10.1109/CVPR.2017.243 http://dx.doi.org/10.1109/CVPR.2017.243 ]
Hui Z, Wang X M and Gao X B. 2018. Fast and accurate single image super-resolution via information distillation network//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 723-731[ DOI: 10.1109/CVPR.2018.00082 http://dx.doi.org/10.1109/CVPR.2018.00082 ]
Kim J, Lee J K and Lee K M. 2016a. Deeply-recursive convolutional network for image super-resolution//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 1637-1645[ DOI: 10.1109/CVPR.2016.181 http://dx.doi.org/10.1109/CVPR.2016.181 ]
Kim J, Lee J K and Lee K M. 2016b. Accurate image super-resolution using very deep convolutional networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 1646-1654[ DOI: 10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ]
Lai W S, Huang J B, Ahuja N and Yang M H. 2017. Deep Laplacian pyramid networks for fast and accurate super-resolution//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5835-5843[ DOI: 10.1109/CVPR.2017.618 http://dx.doi.org/10.1109/CVPR.2017.618 ]
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z H and Shi W Z. 2017. Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of 2017 IEEE Conference on ComputerVision and Pattern Recognition. Honolulu, USA: IEEE: 105-114[ DOI: 10.1109/CVPR.2017.19 http://dx.doi.org/10.1109/CVPR.2017.19 ]
Li L Y, Su Z, Shi X H, Huang E B and Luo X N. 2018. Mutual-detail convolution model for image super-resolution reconstruction. Journal of Image and Graphics, 23(4): 572-582
李浪宇, 苏卓, 石晓红, 黄恩博, 罗笑南. 2018. 图像超分辨率重建中的细节互补卷积模型. 中国图象图形学报, 23(4): 572-582 [DOI:10.11834/jig.170361]
Li Z, Yang J L, Liu Z, Yang X M, Jeon G and Wu W. 2019. Feedback network for image super-resolution//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 3862-3871[ DOI: 10.1109/CVPR.2019.00399 http://dx.doi.org/10.1109/CVPR.2019.00399 ]
Qiu Y J, Wang R X, Tao D P and Cheng J. 2019. Embedded block residual network: a recursive restoration model for single-image super-resolution//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 4179-4188[ DOI: 10.1109/ICCV.2019.00428 http://dx.doi.org/10.1109/ICCV.2019.00428 ]
Shen M Y, Yu P F, Wang R G, Yang J and Xue L X. 2019. Image super-resolution reconstruction via deep network based on multi-staged fusion. Journal of Image and Graphics, 24(8): 1258-1269
沈明玉, 俞鹏飞, 汪荣贵, 杨娟, 薛丽霞. 2019. 多阶段融合网络的图像超分辨率重建. 中国图象图形学报, 24(8): 1258-1269 [DOI:10.11834/jig.180619]
Shi W Z, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D and Wang Z H. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 1874-1883[ DOI: 10.1109/CVPR.2016.207 http://dx.doi.org/10.1109/CVPR.2016.207 ]
Yang J, Li W J, Wang R G and Xue L X. 2019. Generative adversarial network for image super-resolution combining perceptual loss. Journal of Image and Graphics, 24(8): 1270-1282
杨娟, 李文静, 汪荣贵, 薛丽霞. 2019. 融合感知损失的生成式对抗超分辨率算法. 中国图象图形学报, 24(8): 1270-1282 [DOI:10.11834/jig.180613]
Yang X, Mei H Y, Zhang J Q, Xu K, Yin B C, Zhang Q and Wei X P. 2019. DRFN: deep recurrent fusion network for single-image super-resolution with large factors. IEEE Transactions on Multimedia, 21(2): 328-337[DOI:10.1109/tmm.2018.2863602]
Zhang K, Zuo W M and Zhang L. 2019a. Deep plug-and-play super-resolution for arbitrary blur kernels//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 1671-1681[ DOI: 10.1109/CVPR.2019.00177 http://dx.doi.org/10.1109/CVPR.2019.00177 ]
Zhang Z D, Wang X R and Jung C. 2019b. DCSR: dilated convolutions for single image super-resolution. IEEE Transactions on Image Processing, 28(4): 1625-1635[DOI:10.1109/TIP.2018.2877483]
Zhou F, Yang W M and Liao Q M. 2012.Interpolation-based image super-resolution using multisurface fitting. IEEE Transactions on Image Processing, 21(7): 3312-3318[DOI:10.1109/TIP.2012.2189576]
相关作者
相关机构
京公网安备11010802024621
