浏览全部资源
扫码关注微信
收稿：2017-05-08，
修回：2017-10-24，
纸质出版：2018-02-16
移动端阅览
目的
2
克服传统遥感图像超分辨率重建方法依赖同一场景多时相图像序列且需预先配准等缺点，解决学习法中训练效率低和过拟合问题，同时削弱插值操作后的块效应，增强单幅遥感图像超分辨率重建效果。
方法
2
首先构造基于四层卷积的深度神经网络结构，并在结构中前三层卷积后添加参数修正线性单元层和局部响应归一化层进行优化，经过训练得到遥感图像超分辨率重建模型，其次，对多波段遥感图像的亮度空间进行双三次插值，然后使用该模型对插值结果进行重建，并在亮度空间重建结果指导下，使用联合双边滤波来提升其色度空间边缘细节。
结果
2
应用该方法对实验遥感图像进行2倍、3倍、4倍重建时在无参考指标上均优于对比方法，平均清晰度提升约2.5个单位，同时取得了较好的全参考评价结果，在2倍重建时峰值信噪比较传统插值法提升了约2 dB，且平均训练效率较其他学习法提升3倍以上，所得遥感图像重建结果在目视效果上更加细致、自然。
结论
2
实验结果表明，本文设计的网络抗过拟合能力强、训练效率高，重建时针对单幅遥感图像，无需依赖图像序列且不受波段影响，重建结果细节表现较好，具有较强的普适性。
Objective
2
Super-resolution (SR)
which restores a high-resolution (HR) image from single or sequential low-resolution (LR) images
is a widely applied technology in image processing
especially in the remote sensing field. HR remote sensing images are increasingly sought with the rapid advancement of remote sensing technology in agriculture and forestry monitoring
urban planning
and military reconnaissance. However
traditional interpolation-based methods cannot achieve a satisfying effect
while reconstruction-based methods require pre-registration and are constrained by the lack of sequential images. In several modern learning-based methods
complicated network
considerable training time
and neglect of chrominance space still require improvement. To solve these problems
a novel SR method combined with deep learning is proposed in this paper to achieve high-quality SR reconstruction of single remote sensing image
thereby overcoming traditional drawbacks
such as dependence on image sequences or registration. The proposed method also aims to improve the efficiency and reduce the overfitting risk during training and provide a reference for the weakening block effect of chrominance interpolation.
Method
2
The proposed SR reconstruction process is conducted from the luminance and chrominance spaces of single remote sensing image. First
a network model named PL-CNN that is based on a four-layer convolutional neural network (CNN) is optimized with parametric rectified linear unit (PReLU) and local response normalization (LRN) layers considering the autocorrelation and texture richness of remote sensing images. In the PL-CNN
the first to the fourth convolutional layers can successively achieve feature extraction
enhancement
nonlinear mapping
and reconstruction. The deployment of PReLU can accelerate the training speed and retain the image features simultaneously. The LRN layers are used to avoid overfitting
thereby enhancing the final SR effect further. Then
the proposed PL-CNN with an iteration of 2.5 million is trained with an upscaling factor to obtain the SR model by taking the mean square error as the loss function. The training data from the UC Merced land use dataset
with a 0.3 m resolution
thereby covering 21 categories of remote sensing scenes. The training inputs are used to simulate the LR remote sensing image patches
and the outputs correspond to the original HR remote sensing images. For multiband images
the model is utilized to obtain a reconstructed result in the luminance space. Then
a joint bilateral filtering with a pixel scope of 3×3 under the guidance of the result is introduced to improve the edge details of the chrominance space after bicubic interpolation. A single-band image could be considered a special case of multiband image in which its reconstruction excludes the chrominance part.
Result
2
A series of simulation experiments is conducted to verify the validity and applicability of the proposed SR method
and a dataset (RS5) that includes five remote sensing images with different sizes and resolutions is established to serve as the experimental images. Full-and no-reference evaluations are applied to value the quality of the SR reconstructed images objectively and fairly. Full-reference evaluation indexes include peak signal-to-noise ratio (PSNR) and structure similarity index (SSIM)
while the no-reference evaluation indexes include spatial and spectral entropies (SSEQ) and clarity. Results show that the proposed reconstruction of RS5 is superior to others at no-reference evaluation indexes with upscaling factors of 2
3
and 4. The SSEQ is enhanced
and the mean clarity value improves by 2.5 standard units. The proposed method's results also display advantageous PSNR and efficiency
thereby achieving 2 dB better in PSNR than in bicubic interpolation algorithm and limiting the average training time to one-third or less than the other learning-based methods. The visualization of the first-layer filters is rich in textures
and the typical feature maps are gradually enhanced along with the layers. The capability of joint bilateral filtering to remove the block effect and sharpen the edges is easily verified by observing the images of the chrominance space before and after filtering. Furthermore
the PSNR result continuously improves with the increase in iteration
thereby indicating a potential ameliorated orientation. A Landsat-8 image of Tangshan
China is selected for reconstruction through the PL-CNN method and decomposition into red
green
and blue bands to verify the band applicability of the proposed method. The PSNR result for each band is more than 28 dB
and the average SSIM is approximately 98.5%. The mean value and standard deviation of the original and reconstructed images in the three bands are near
thus manifesting that the proposed method is unrestricted to band factors and has a robust applicability.
Conclusion
2
A SR reconstruction method of single remote sensing image combined with deep learning is proposed. The optimized network
namely
PL-CNN
on the basis the CNN extracts additional features and performs well in terms of anti-overfitting. Moreover
the PReLU structure can effectively accelerate the training process. Experimental results suggest that the proposed method is unrestricted to the image sequence or band
thereby aiming for a single remote sensing image and considering the chrominance space
and the reconstruction quality under several upscaling factors provides evident advantages over the traditional SR reconstruction methods. Owing to the natural and clear visual effect of images reconstructed with PL-CNN
the method has broad prospects
especially in the remote sensing field. Future studies may be conducted using additional samples
appropriately increasing the iterations
and focusing on high upscaling factors.
Harris J L. Diffraction and resolving power[J]. Journal of the Optical Society of America (1917-1983), 1964, 54(7):931-933.[DOI:10.1364/JOSA.54.000931]
Tsai R, Huang T. Multiple frame image restoration and registration[M]//Advances in Computer Vision&Image Processing. Greenwich, Ct: Jai Press Inc., 1984: 317-339.
Hardie R C, Barnard K J, Armstrong E E. Joint MAP registration and high-resolution image estimation using a sequence of undersampled images[J]. IEEE Transactions on Image Processing, 1997, 6(12):1621-1633.[DOI:10.1109/83.650116]
Tian J, Ma K K. Stochastic super-resolution image reconstruction[J]. Journal of Visual Communication and Image Representation, 2010, 21(3):232-244.[DOI:10.1016/j.jvcir.2010.01.001]
Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(6):1127-1133.[DOI:10.1109/TPAMI.2010.25]
Zhang Y, Xu K, Li Y. Remote sensing image super-resolution based on POCS and out-of-core[J]. Journal of Tsinghua University, 2010, 50(10):1743-1746.
张砚, 徐昆, 李勇.基于外存和凸集投影法的遥感图像超分辨率方法[J].清华大学学报:自然科学版, 2010, 50(10):1743-1746
Chang H, Yeung D Y, Xiong Y. Super-resolution through neighbor embedding[C]//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC, USA: IEEE, 2004: 275-282. [ DOI:10.1109/CVPR.2004.1315043 http://dx.doi.org/10.1109/CVPR.2004.1315043 ]
Yang J C, Wright J, HuangT S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873.[DOI:10.1109/TIP.2010.2050625]
Jin W, Fu R D, Ye M. Nephogram super-resolution algorithm using over-complete dictionary via sparse representation[J]. Journal of Remote Sensing, 2012, 16(2):275-285.
金炜, 符冉迪, 叶明.过完备字典稀疏表示的云图超分辨率算法[J].遥感学报, 2012, 16(2):275-285. [DOI:10.11834/jrs.20121021]
Timofte R, De Smet V, Van Gool L. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE, 2013: 1920-1927. [ DOI:10.1109/ICCV.2013.241 http://dx.doi.org/DOI:10.1109/ICCV.2013.241 ]
Dong C, Loy C C, He K M, et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of European Conference on Computer Vision. Cham: Springer, 2014: 184-199. [ DOI:10.1007/978-3-319-10593-2_13 http://dx.doi.org/10.1007/978-3-319-10593-2_13 ]
Dong C, Loy C C, Tang X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of European Conference on Computer Vision. Cham: Springer, 2016: 391-407. [ DOI:10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25 ]
Xu R, Zhang J G, Huang K Q. Image super-resolution using two-channel convolutional neural networks[J]. Journal of Image and Graphics, 2016, 21(5):556-564.
徐冉, 张俊格, 黄凯奇.利用双通道卷积神经网络的图像超分辨率算法[J].中国图象图形学报, 2016, 21(5):556-564. [DOI:10.11834/jig.20160503]
Lécun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.[DOI:10.1109/5.726791]
Song Y, Hong X H, McLoughlin I, et al. Image classification with CNN-based fisher vector coding[C]//Proceedings of 2016 Visual Communications and Image Processing. Chengdu, China: IEEE, 2016: 1-4. [ DOI:10.1109/VCIP.2016.7805494 http://dx.doi.org/10.1109/VCIP.2016.7805494 ]
Bappy J H, Roy-Chowdhury A K. CNN based region proposals for efficient object detection[C]//Proceedings of 2016 IEEE International Conference on Image Processing. Phoenix, AZ, USA: IEEE, 2016: 3658-3662. [ DOI:10.1109/ICIP.2016.7533042 http://dx.doi.org/10.1109/ICIP.2016.7533042 ]
Zhang B, Quan C Q, Ren F J. Study on CNN in the recognition of emotion in audio and images[C]//Proceedings of 2016 IEEE/ACIS 15th International Conference on Computer and Information Science. Okayama, Japan: IEEE, 2016: 1-5. [ DOI:10.1109/ICIS.2016.7550778 http://dx.doi.org/10.1109/ICIS.2016.7550778 ]
He K M, Zhang X X, Ren S Q, et al. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 1026-1034. [ DOI:10.1109/ICCV.2015.123 http://dx.doi.org/10.1109/ICCV.2015.123 ]
Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada: Curran Associates Inc., 2012: 1097-1105. [ DOI:10.1145/3065386 http://dx.doi.org/10.1145/3065386 ]
Kopf J, Cohen M F, Lischinski D, et al. Joint bilateral upsampling[J]. ACM Transactions on Graphics, 2007, 26(3):#96.[DOI:10.1145/1276377.1276497]
Tomasi C, Manduchi R. Bilateral filtering for gray and color images[C]//Proceedings of the sixth International Conference on Computer Vision. Bombay, India: IEEE, 1998: 839-846. [ DOI:10.1109/ICCV.1998.710815 http://dx.doi.org/10.1109/ICCV.1998.710815 ]
Yang Y, Newsam S. Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems. New York, USA: ACM, 2010: 270-279. [ DOI:10.1145/1869790.1869829 http://dx.doi.org/10.1145/1869790.1869829 ]
Wang H H, Peng J X, Wu W, et al. A study of evaluation methods on performance of the multi-source remote sensing image fusion[J]. Computer Engineering and Applications, 2003, 39(25):33-37.
王海晖, 彭嘉雄, 吴巍, 等.多源遥感图像融合效果评价方法研究[J].计算机工程与应用, 2003, 39(25):33-37. [DOI:10.3321/j.issn:1002-8331.2003.25.010]
Liu L X, Liu B, Huang H, et al. No-reference image quality assessment based on spatial and spectral entropies[J]. Signal Processing:Image Communication, 2014, 29(8):856-863.[DOI:10.1016/j.image.2014.06.006]
Li L, Luo H, Tang X M, et al. Characteristic analysis and quality assessment of ZY-3 multi-spectral image[J]. Remote Sensing for Land&Resources, 2014, 26(1):17-24.
李霖, 罗恒, 唐新明, 等.资源三号卫星多光谱图像特征分析和质量评价[J].国土资源遥感, 2014, 26(1):17-24. [DOI:10.6046/gtzyyg.2014.01.04]
Xia G S, Hu J W, Hu F, et al. AID:a benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7):3965-3981.[DOI:10.1109/TGRS.2017.2685945]
相关作者
相关机构
京公网安备11010802024621
