浏览全部资源
扫码关注微信
Published： 16 January 2023 ，
Accepted： 05 October 2022
移动端阅览
Liangjian Deng, Ran Ran, Xiao Wu, Tianjing Zhang. CNN-based remote sensing pan-sharpening: a critical review. [J]. Journal of Image and Graphics 28(1):57-79(2023)
随着计算机科学、遥感科学和大数据科学等领域的迅速发展，基于卷积神经网络的方法在图像处理、计算机视觉等任务上发挥着越来越重要的作用。而在遥感图像全色锐化领域，卷积神经网络由于其优秀的融合效果，已得到研究学者的广泛关注并有大量的研究成果产生。尽管如此，依然有一些亟待解决的问题，例如缺乏全色锐化数据集的仿真细节描述、公平公开的训练—测试数据集、简单易懂的统一代码编写框架等。对此，本文主要从以下几方面回顾当前遥感图像全色锐化问题在卷积神经网络方面的一些进展，并针对前述问题发布相关数据集和代码编写框架。1)详细介绍7种典型的基于卷积神经网络的全色锐化方法，并在统一数据集上进行公平比较(包括与典型传统方法的比较)；2)详细介绍训练—测试数据集的仿真细节，并发布相关卫星(如WorldView-3，QuickBird，GaoFen2，WorldView-2)的全色锐化训练—测试数据集；3)针对本文介绍的7种基于卷积神经网络的方法，发布基于Pytorch深度学习库的Python代码统一编写框架，便于后来初学者入门、开展研究以及公平比较；4)发布统一的全色锐化传统—深度学习方法MATLAB测试软件包，便于后来学者进行公平的实验测试对比；5)对本领域的未来研究方向进行讨论和展望。本文的相关数据集和代码详见课题主页：
https://liangjiandeng.github.io/PanCollection.html
https://liangjiandeng.github.io/PanCollection.html
。
The aim of pan-sharpening is focused on data acquisition and processing captured from multiple remote sensing satellites in terms of machine learning and signal processing techniques. To produce a high-qu
ality multispectral image
its objective is oriented to fuse a low spatial resolution-based multispectral (MS) image and a high spatial resolution-based panchromatic (PAN) image. Machine learning (ML) technique is being developed to deal with it. Pan-sharpening
a recent multispectral image-based image fusion technique
has been concerned about in terms of machine learning
which is featured by its medium/low spatial resolution. In recent years
remote sensing (RS) science
big data science and convolutional neural networks based (CNNs-based) techniques (a sub-category of ML)
have promoted image processing
computer vision and its contexts. Thanks to the priority of RS images-based pan-sharpening of fusion
CNN-oriented researches have been developing dramatically
but some challenging issues are required to be handled
such as the detailed introduction to typical pan-sharpening CNNs
data simulation
feasible and open training-test datasets
simple and easy-to-understand unified code writing framework
etc. So
we mainly review the growth of CNNs for RS-based pan-sharpening from 5 aspects as following: 1) detailed introduction of 7 CNN-based pan-sharpening methods and comparative analysis are given out under the datasets-coordinated; 2) the simulation of training-test datasets is introduced in details. We intend to release the pan-sharpening datasets of related satellites (such as WorldView-3
QuickBird
GaoFen2
WorldView-2 satellites); 3) for all 7 CNN-based methods introduced
Python codes-based on Pytorch library is released in a framework-integrated to facilitate the completeness and feasibility further; 4) a pan-sharpening-unified MATLAB software package is demonstrated for the test of deep-learning and traditional approaches
which is beneficial to conduct balancing tests; and 5) future research direction is predicted as well. Please click the link for project details:
https://liangjiandeng.github.io/PanCollection.html
https://liangjiandeng.github.io/PanCollection.html
. First
CNN-based techniques have proven to have good performance under the similar situations. But
the performance has decreased when a complete different situation is employed (such as in contrast with existing contemporary methods)
a limitation of these approaches are challenged. Next
the fine-tuning technique has demonstrated its effectiveness in resolving the problem mentioned above
ensuring better performance for these complicated test scenarios. Compared to the quickest speed conventional methods
the computing load of the studied CNN-based algorithms can be determined as an appropriate manner during the testing phase. Finally
we recommend that a novel design of CNN-based pan-sharpening methods. The skip connection operation can aid ML-based methods in achieving a faster convergence when it is focused on the analyzed CNN-based pan-sharpening approaches. Instead
a stronger feature extraction and learning can be supported by the design of multi-scaled architectures (including the bidirectional structure even). Furthermore
the capacity of the generalized networks can be improved using a fine-tuning technique and learning in a customized domain (instead of the original image domain). However
there are still some problems to be resolved. The computational weight problem is challenged to create networks with fewer parameters so as to guarantee network performance (even achieving a quick convergence). Furthermore
the recent advancements in CNN for pan-sharpening have a restricted capacity for generalization. A lower resolution-based initial vision is beneficial to provide labels for network training. As a result
loss functions-based techniques (unsupervised) have been developed in evaluating similarities at full resolution. The future research is potential to be developed for full resolution further.
全色锐化卷积神经网络(CNN)典型卷积神经网络方法比较数据集发布代码框架发布全色锐化综述
pansharpeningconvolutional neural network(CNN)comparison of typical CNN methodsdatasets releasingcoding-framework releasingpansharpening survey
Aiazzi B, Alparone L, Baronti S and Garzelli A. 2002. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing, 40(10): 2300-2312 [DOI: 10.1109/TGRS.2002.803623]
Arienzo A, Vivone G, Garzelli A, Alparone L and Chanussot J. 2022. Full-resolution quality assessment of pansharpening: theoretical and hands-on approaches. IEEE Geoscience and Remote Sensing Magazine, 10(3): 168-201 [DOI: 10.1109/MGRS.2022.3170092]
Berné O, Helens A, Pilleri P and Joblin C. 2010. Non-negative matrix factorization pansharpening of hyperspectral data: an application to mid-infrared astronomy//Proceedings of 2010 Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. Reykjavik, Iceland: IEEE: 1-4 [DOI: 10.1109/WHISPERS.2010.5594900http://dx.doi.org/10.1109/WHISPERS.2010.5594900]
Buades A, Coll B, Duran J and Sbert C. 2014. Implementation of nonlocal pansharpening image fusion. Image Processing On Line, 4: 1-15 [DOI: 10.5201/ipol.2014.98]
Burt P J and Adelson E H. 1987. The laplacian pyramid as a compact image code. Readings in Computer Vision, 31(4): 671-679 [DOI: 10.1016/B978-0-08-051581-6.50065-9]
Carper W J, Lillesand T M and Kiefer R W. 1990. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data. Photogrammetric Engineering and Remote Sensing, 56(4): 459-467
Chavez P S and Kwarteng A Y. 1989. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing, 55(3): 339-348
Cheng M, Wang C and Li J. 2014. Sparse representation based pansharpening using trained dictionary. IEEE Geoscience and Remote Sensing Letters, 11(1): 293-297 [DOI: 10.1109/LGRS.2013.2256875]
Choi J, Yu K Y and Kim Y. 2011. A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1): 295-309 [DOI: 10.1109/TGRS.2010.2051674]
Deng L J, Feng M Y and Tai X C. 2019. The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior. Information Fusion, 52: 76-89 [DOI: 10.1016/j.inffus.2018.11.014]
Deng L J, Vivone G, Jin C and Chanussot J. 2021. Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(8): 6995-7010 [DOI: 10.1109/TGRS.2020.3031366]
Deng L J, Vivone G, Paoletti M E, Scarpa G, He J, Zhang Y J, Chanussot J and Plaza A J. 2022. Machine learning in pansharpening: a benchmark, from shallow to deep networks. IEEE Geoscience and Remote Sensing Magazine, 2-38 [DOI: 10.1109/MGRS.2022.3187652]
Dian R W, Li S T and Kang X D. 2021. Regularizing hyperspectral and multispectral image fusion by CNN denoiser. IEEE Transactions on Neural Networks and Learning Systems, 32(3): 1124-1135 [DOI: 10.1109/TNNLS.2020.2980398]
Do M N and Vetterli M. 2005. The contourlet transform: an efficient directional multiresolutionimage representation. IEEE Transactions on Image Processing, 14(12): 2091-2106 [DOI: 10.1109/TIP.2005.859376]
Dong C, Loy C C, He K M and Tang X O. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307 [DOI: 10.1109/TPAMI.2015.2439281]
Fang F M, Li F, Shen C M and Zhang G X. 2013. A variational approach for pan-sharpening. IEEE Transactions on Image Processing, 22(7): 2822-2834 [DOI: 10.1109/TIP.2013.2258355]
Feng Y Q, Liu J M, Chen K, Wang B and Zhao Z X. 2022. Optimization algorithm unfolding deep networks of detail injection model for pansharpening. IEEE Geoscience and Remote Sensing Letters, 19: 1-5 [DOI: 10.1109/LGRS.2021.3077183]
Hardie R C, Eismann M T and Wilson G L. 2004. MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor. IEEE Transactions on Image Processing, 13(9): 1174-1184 [DOI: 10.1109/TIP.2004.829779]
He L, Rao Y Z, Li J, Chanussot J, Plaza A, Zhu J W and Li B. 2019. Pansharpening via detail injection based convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(4): 1188-1204 [DOI: 10.1109/JSTARS.2019.2898574]
Huang W, Xiao L, Wei Z H, Liu H Y and Tang S Z. 2015. A new pan-sharpening method with deep neural networks. IEEE Geoscience and Remote Sensing Letters, 12(5): 1037-1041 [DOI: 10.1109/LGRS.2014.2376034]
Hui T W, Loy C C and Tang X O. 2016. Depth map super-resolution by deep multi-scale guidance//Proceedings of the 14th European Conference on Computer Vision. Cham, Switzerland: Springer: 353-369 [DOI: 10.1007/978-3-319-46487-9_22http://dx.doi.org/10.1007/978-3-319-46487-9_22]
Jin Z R, Zhang T J, Jiang T X, Vivone G and Deng L J. 2022. LAGConv: local-context adaptive convolution kernels with global harmonic bias for pansharpening//Proceedings of the 36th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press: 1-9 [DOI: 10.1609/aaai.v36i1.19996http://dx.doi.org/10.1609/aaai.v36i1.19996]
Kawakami R, Matsushita Y, Wright J, Ben-Ezra M, Tai Y W and Ikeuchi K. 2011. High-resolution hyperspectral imaging via matrix factorization//Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, USA: IEEE: 2329-2336 [DOI: 10.1109/CVPR.2011.5995457http://dx.doi.org/10.1109/CVPR.2011.5995457]
Laben C A and Brower B V. 2000. US 6011875 Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. Rochester, USA: Eastman Kodak Company
Li S T, Yin H T and Fang L Y. 2013. Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Transactions on Geoscience and Remote Sensing, 51(9): 4779-4789 [DOI: 10.1109/TGRS.2012.2230332]
Liu Q J, Zhou H Y, Xu Q Z, Liu X Y and Wang Y H. 2021. PSGAN: a generative adversarial network for remote sensing image pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(12): 10227-10242 [DOI: 10.1109/TGRS.2020.3042974]
Lolli S, Alparone L, Garzelli A and Vivone G. 2017. Haze correction for contrast-based multispectral pansharpening. IEEE Geoscience and Remote Sensing Letters, 14(12): 2255-2259 [DOI: 10.1109/LGRS.2017.2761021]
Luo S Y, Zhou S B, Feng Y and Xie J G. 2020. Pansharpening via unsupervised convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13: 4295-4310 [DOI: 10.1109/JSTARS.2020.3008047]
Ma J Y, Yu W, Chen C, Liang P W, Guo X J and Jiang J J. 2020. Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion, 62: 110-120 [DOI: 10.1016/j.inffus.2020.04.006]
Mallat S G. 1989. A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7): 674-693 [DOI: 10.1109/34.192463]
Masi G, Cozzolino D, Verdoliva L and Scarpa G. 2016. Pansharpening by convolutional neural networks. Remote Sensing, 8(7): #594 [DOI: 10.3390/rs8070594]
Molina R, Vega M, Mateos J and Katsaggelos A K. 2008. Variational posterior distribution approximation in Bayesian super resolution reconstruction of multispectral images. Applied and Computational Harmonic Analysis, 24(2): 251-267 [DOI: 10.1016/j.acha.2007.03.006]
Nason G P and Silverman B W. 1995. The stationary wavelet transform and some statistical applications//Antoniadis A, Oppenheim G, eds. Wavelets and Statistics. New York, USA: Springer. [DOI: 10.1007/978-1-4612-2544-7_17http://dx.doi.org/10.1007/978-1-4612-2544-7_17]
Palsson F, Sveinsson J R and Ulfarsson M O. 2014. A new pansharpening algorithm based on total variation. IEEE Geoscience and Remote Sensing Letters, 11(1): 318-322 [DOI: 10.1109/LGRS.2013.2257669]
Qu Y, Baghbaderani R K, Qi H R and Kwan C. 2021. Unsupervised pansharpening based on self-attention mechanism. IEEE Transactions on Geoscience and Remote Sensing, 59(4): 3192-3208 [DOI: 10.1109/TGRS.2020.3009207]
Shah V P, Younan N H and King R L. 2008. An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Transactions on Geoscience and Remote Sensing, 46(5): 1323-1335 [DOI: 10.1109/TGRS.2008.916211]
Shen H F, Jiang M H, Li J, Yuan Q Q, Wei Y C and Zhang L P. 2019. Spatial-spectral fusion by combining deep learning and variational model. IEEE Transactions on Geoscience and Remote Sensing, 57(8): 6169-6181 [DOI:10.1109/TGRS.2019.2904659]
Shettigara V K. 1992. A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set. Photogrammetric Engineering and Remote Sensing, 58(5): 561-567
Teodoro A, Bioucas-Dias J M and Figueiredo M. 2017. Sharpening hyperspectral images using plug-and-play priors//Proceeding of the 13th International Conference on Latent Variable Analysis and Signal Separation. Cham, Switzerland: Springer: 392-402 [DOI: 10.1007/978-3-319-53547-0http://dx.doi.org/10.1007/978-3-319-53547-0]
Tu T M, Su S C, Shyu H C and Huang P S. 2001. A new look at IHS-like image fusion methods. Information Fusion, 2(3): 177-186 [DOI: 10.1016/S1566-2535(01)00036-7]
Vivone G. 2019. Robust band-dependent spatial-detail approaches for panchromatic sharpening. IEEE Transactions on Geoscience and Remote Sensing, 57(9): 6421-6433 [DOI: 10.1109/TGRS.2019.2906073]
Vivone G, Mura M D, Garzelli A, Restaino R, Scarpa G, Ulfarsson M O, Alparone L and Chanussot J. 2021. A new benchmark based on recent advances in multispectral pansharpening: revisiting pansharpening with classical and emerging pansharpening methods. IEEE Geoscience and Remote Sensing Magazine, 9(1): 53-81 [DOI: 10.1109/MGRS.2020.3019315]
Vivone G, Restaino Rand Chanussot J. 2018a. A regression-based high-pass modulation pansharpening approach. IEEE Transactions on Geoscience and Remote Sensing, 56(2): 984-996 [DOI: 10.1109/TGRS.2017.2757508]
Vivone G, Restaino R and Chanussot J. 2018b. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Transactions on Image Processing, 27(7): 3418-3431 [DOI: 10.1109/TIP.2018.2819501]
Vivone G, Simões M, Mura M D, Restaino R, Bioucas-Dias J M, Licciardi G A and Chanussot J. 2015. Pansharpening based on semiblind deconvolution. IEEE Transactions on Geoscience and Remote Sensing, 53(4): 1997-2010 [DOI: 10.1109/TGRS.2014.2351754]
Wang L and Yang X M. 2021. Remote sensing image pansharpening feedback network based on perceptual loss. Computer Science, 48(8): 91-98
王乐, 杨晓敏. 2021. 基于感知损失的遥感图像全色锐化反馈网络. 计算机科学, 48(8): 91-98 [DOI: 10.11896/jsjkx.200700112]
Wei Y C, Yuan Q Q, Shen H F and Zhang L P. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10): 1795-1799 [DOI: 10.1109/LGRS.2017.2736020]
Wu Z C, Huang T Z, Deng L J, Hu J F and Vivone G. 2022. VO+net: an adaptive approach using variational optimization and deep learning for panchromatic sharpening. IEEE Transactions on Geoscience and Remote Sensing, 60: 1-16 [DOI: 10.1109/TGRS.2021.3066425]
Wu Z C, Huang T Z, Deng L J, Vivone G, Miao J Q, Hu J F and Zhao X L. 2020. A new variational approach based on proximal deep injection and gradient intensity similarity for spatio-spectral image fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13: 6277-6290 [DOI: 10.1109/JSTARS.2020.3030129]
Xiao L, Liu P F and Li H. 2020. Progress and challenges in the fusion of multisource spatial-spectral remote sensing images. Journal of Image and Graphics, 25(5): 851-863
肖亮, 刘鹏飞, 李恒. 2020. 多源空—谱遥感图像融合方法进展与挑战. 中国图象图形学报, 25(5): 851-863 [DOI: 10.11834/jig.190620]
Xie W Y, Lei J, Cui Y H, Li Y S and Du Q. 2020. Hyperspectral pansharpening with deep priors. IEEE Transactions on Neural Networks and Learning Systems, 31(5): 1529-1543 [DOI: 10.1109/TNNLS.2019.2920857]
Xu S, Zhang J S, Zhao Z X, Sun K, Liu J M and Zhang C X. 2021. Deep gradient projection networks for pan-sharpening//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE: 1366-1375 [DOI: 10.1109/CVPR46437.2021.00142http://dx.doi.org/10.1109/CVPR46437.2021.00142]
Yang J F, Fu X Y, Hu Y W, Huang Y, Ding X H and Paisley J. 2017. PanNet: a deep network architecture for pan-sharpening//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 5449-5457
Yang Y, Su Z, Huang S Y, Wan W G, Tu W and Lu H Y. 2022. Survey of deep-learning approaches for pixel-level pansharpening. National Remote Sensing Bulletin, 26(12): 2411-2432
杨勇, 苏昭, 黄淑英, 万伟国, 涂伟, 卢航远. 2022. 基于深度学习的像素级全色图像锐化研究综述. 遥感学报, 26(12): 2411-2432 [DOI: 10.11834/jrs.20211325]
Yokoya N, Yairi T and Iwasaki A. 2012. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2): 528-537 [DOI: 10.1109/TGRS.2011.2161320]
Yuan Q Q, Wei Y C, Meng X C, Shen H F and Zhang L P. 2018. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3): 978-989 [DOI: 10.1109/JSTARS.2018.2794888]
Zhang L P and Zhang L F. 2011. Hyperspectral Remote Sensing. Beijing: Surveying and Mapping Press
张良培, 张立福. 2011. 高光谱遥感. 北京: 测绘出版社
Zhang T J, Deng L J, Huang T Z, Chanussot J and Vivone G. 2022. A triple-double convolutional neural network for panchromatic sharpening. IEEE Transactions on Neural Networks and Learning Systems: 1-14 [DOI: 10.1109/TNNLS.2022.3155655]
Zhang Y F, De Backer S and Scheunders P. 2009. Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing, 47(11): 3834-3843 [DOI: 10.1109/TGRS.2009.2017737]
Zhang Y J, Liu C, Sun M W and Ou Y J. 2019. Pan-sharpening using an efficient bidirectional pyramid network. IEEE Transactions on Geoscience and Remote Sensing, 57(8): 5549-5563 [DOI: 10.1109/TGRS.2019.2900419]
Zhong J Y, Yang B, Huang G Y, Zhong F and Chen Z Z. 2016. Remote sensing image fusion with convolutional neural network. Sensing and Imaging, 17(1): #10 [DOI: 10.1007/s11220-016-0135-6]
Zhu X X, Grohnfeldt C and Bamler R. 2016. Exploiting joint sparsity for pansharpening: the J-SparseFI algorithm. IEEE Transactions on Geoscience and Remote Sensing, 54(5): 2664-2681 [DOI: 10.1109/TGRS.2015.2504261]
Zhuo Y W, Zhang T J, Hu J F, Dou H X, Huang T Z and Deng L J. 2022. A deep-shallow fusion network with multidetail extractor and spectral attention for hyperspectral pansharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15: 7539-7555 [DOI: 10.1109/JSTARS.2022.3202866]
相关文章
相关作者
相关机构
京公网安备11010802024621