多源空—谱遥感图像融合方法进展与挑战

肖亮; 刘鹏飞; 李恒

doi:10.11834/jig.190620

学者观点 | 浏览量 : 0 下载量: 31 CSCD: 11

PDF
导出
分享
收藏
专辑

多源空—谱遥感图像融合方法进展与挑战
Progress and challenges in the fusion of multisource spatial-spectral remote sensing images
2020年25卷第5期页码：851-863
收稿：2019-11-27，

修回：2019-12-24，

录用：2019-12-31，

纸质出版：2020-05-16
DOI： 10.11834/jig.190620
稿件说明：

移动端阅览

肖亮, 刘鹏飞, 李恒. 多源空—谱遥感图像融合方法进展与挑战[J]. 中国图象图形学报, 2020,25(5):851-863. DOI： 10.11834/jig.190620.

Liang Xiao, Pengfei Liu, Heng Li. Progress and challenges in the fusion of multisource spatial-spectral remote sensing images[J]. Journal of Image and Graphics, 2020, 25(5): 851-863. DOI： 10.11834/jig.190620.

摘要

多源空—谱遥感图像融合方法作为两路不完全观测多通道数据的计算重构反问题，其挑战在于补充信息不足、模糊和噪声等引起的病态性，现有方法在互补特征保持的分辨率增强方面仍有很大的改进空间。为了推动遥感图像融合技术的发展，本文系统概述目前融合建模的代表性方法，包括成分替代、多分辨率分析、变量回归、贝叶斯、变分和模型与数据混合驱动等方法体系及其存在问题。从贝叶斯融合建模的角度，分析了互补特征保真和图像先验在优化融合中的关键作用和建模机理，并综述了目前若干图像先验建模的新趋势，包括：分数阶正则化、非局部正则化、结构化稀疏表示、矩阵低秩至张量低秩表示、解析先验与深度先验的复合等。本文对各领域面临的主要挑战和可能的研究方向进行了概述和讨论，指出解析模型和数据混合驱动将是图像融合的重要发展方向，并需要结合成像退化机理、数据紧致表示和高效计算等问题，突破现有模型优化融合的技术瓶颈，进一步发展更优良的光谱信息保真和更低算法复杂度的融合方法。同时，为了解决大数据问题，有必要在Hadoop和SPARK等大数据平台上进行高性能计算，以更有利于多源数据融合算法的加速实现。

Abstract

With the rapid development of earth observation technology

remote sensing opens up the possibility of multiplatform

multisensor

and multiangle observation

and the acquisition ability of multimodal datasets in a joint manner has been considerably improved. Extensive attention has been given to multisource data fusion because such technology can be used to improve the performance of processing approaches with respect to available applications. In this study

we focus on reviewing state-of-the-art multisource remote image fusion. Three typical problems

namely

pansharpening

hypersharpening

and the fusion of hyperspectral ands multispectral images

have been comprehensively investigated. A mathematical modeling view of many important contributions specifically dedicated to the three topics is provided.First

the major challenge and complex imaging relationship between available multisource spatial-spectral remote sensing images are discussed. As an inverse problem of recovering the latent high-resolution image from two branches of incomplete observed multichannel data

the challenge lies in the ill-posed condition caused by insufficient supplementary information

optical blur

and noise. Therefore

the existing data fusion method still has considerable room for resolution enhancing with complementary information preserving capacity. Second

a comprehensive survey is conducted for the representative mathematical modeling paradigms

including the component substitution scheme

multiresolution analysis framework

Bayesian model

variational model

and data- and model-driven optimization methods

and their existing problems. From the point of view of Bayesian fusion modeling

this study analyzes the key role and modeling mechanism of complementary features

preserving data fidelity and image prior terms in the optimization model. Then

this work summarizes the new trends in image priori modeling

including fractional-order regularization

nonlocal regularization

structured sparse representation

matrix low rank

tensor representation

and compound regularization with analytical and deep priors. Lastly

the major challenges and possible research direction in each area are outlined and discussed. The hybrid analytical model and data-driven framework will be important research directions. Breaking through the technical bottleneck of existing model optimization-based fusion methods with the imaging degradation model

data compact representation

and efficient computation

and developing fusion methods with improved spectral-preserving performance and reduced computational complexity are necessary. To cope with the big data problem

high-performance computing on big data platforms

such as Hadoop and SPARK

will obtain promising applications for multisource data-accelerated-based fusion.

关键词

Keywords

references

Adler J and Öktem O. 2018. Learned primal-dual reconstruction. IEEE Transactions on Medical Imaging, 37(6):1322-1332[DOI:10.1109/TMI.2018.2799231]

Aiazzi B, Alparone L, Baronti S, Garzelli A and Selva M. 2006. MTF-tailored multiscale fusion of high-resolution MS and pan imagery. Photogrammetric Engineering and Remote Sensing, 72(5):591-596[DOI:10.14358/pers.72.5.591]

Aiazzi B, Baronti S, Lotti F and Selva M. 2009. A comparison between global and context-adaptive pansharpening of multispectral images. IEEE Geoscience and Remote Sensing Letters, 6(2):302-306[DOI:10.1109/LGRS.2008.2012003]

Alparone L, Baronti S, Aiazzi B and Garzelli A. 2016. Spatial methods for multispectral pansharpening:multiresolution analysis demystified. IEEE Transactions on Geoscience and Remote Sensing, 54(5):2563-2576[DOI:10.1109/TGRS.2015.2503045]

Ballester C, Caselles V, Igual L, Verdera J and Rougé B. 2006. A variational model for P+XS image fusion. International Journal of Computer Vision, 69(1):43-58[DOI:10.1007/s11263-006-6852-x]

Bioucas-Dias J M, Plaza A, Camps-Valls G, Scheunders P, Nasrabadi N and Chanussot J. 2013. Hyperspectral remote sensing data analysis and future challenges. IEEE Geoscience and Remote Sensing Magazine, 1(2):6-36[DOI:10.1109/MGRS.2013.2244672]

Borgerding M, Schniter P and Rangan S. 2017. AMP-inspired deep networks for sparse linear inverse problems. IEEE Transactions on Signal Processing, 65(16):4293-4308[DOI:10.1109/TSP.2017.2708040]

Chien J T and Bao Y T. 2018. Tensor-factorized neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(5):1998-2011[DOI:10.1109/TNNLS.2017.2690379]

Choi J, Yu K and Kim Y. 2011. A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1):295-309[DOI:10.1109/TGRS.2010.2051674]

Dian R W and Li S T. 2019. Hyperspectral image super-resolution via subspace-based low tensor multi-rank regularization. IEEE Transactions on Image Processing, 28(10):5135-5146[DOI:10.1109/TIP.2019.2916734]

Dian R W, Li S T, Guo A J and Fang L Y. 2018. Deep hyperspectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems, 29(11):5345-5355[DOI:10.1109/TNNLS.2018.2798162]

Dian R W, Li S T and Fang L Y. 2019. Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Transactions on Neural Networks and Learning Systems, 30(9):2672-2683[DOI:10.1109/TNNLS.2018.2885616]

Dong C, Loy C C, He K M and Tang X O. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295-307[DOI:10.1109/TPAMI.2015.2439281]

Dong W S, Fu F Z, Shi G M, Gao X, Wu J J, Li G Y and Li X. 2016. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing, 25(5):2337-2352[DOI:10.1109/TIP.2016.2542360]

Duran J, Buades A, Coll B and Sbert C. 2014. A nonlocal variational model for pansharpening image fusion. SIAM Journal on Imaging Sciences, 7(2):761-796[DOI:10.1137/130928625]

Fang F M, Li F, Shen C M and Zhang G X. 2013. A variational approach for pan-sharpening. IEEE Transactions on Image Processing, 22(7):2822-2834[DOI:10.1109/TIP.2013.2258355]

Garzelli A, Nencini F and Capobianco L. 2008. Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46(1):228-236[DOI:10.1109/TGRS.2007.907604]

González-Audícana M, Otazu X, Fors O and Seco A. 2005. Comparison between Mallat's and the 'Trous' discrete wavelet transform based algorithms for the fusion of multispectral andpanchromatic images. International Journal of Remote Sensing, 26(3):595-614[DOI:10.1080/01431160512331314056]

Gregor K and LeCun Y. 2010. Learning fast approximations of sparse coding//Proceedings of the 27th International Conference on Machine Learning. Omnipress: ACM: 399-406

He X Y, Condat L, Bioucas-Dias J M, Chanussot J and Xia J S. 2014. A new pansharpening method based on spatial and spectral sparsity priors. IEEE Transactions on Image Processing, 23(9):4160-4174[DOI:10.1109/TIP.2014.2333661]

Huang J Z, Zhang T and Metaxas D. 2009. Learning with structured sparsity//Proceedings of the 26th Annual International Conference on Machine Learning. Montreal, Quebec, Canada: ACM: 417-424[ DOI: 10.1145/1553374.1553429 http://dx.doi.org/10.1145/1553374.1553429 ]

Huang W, Xiao L, Wei Z H, Liu H Y and Tang S Z. 2015. A new pan-sharpening method with deep neural networks. IEEE Geoscience and Remote Sensing Letters, 12(5):1037-1041[DOI:10.1109/LGRS.2014.2376034]

Hutchinson B, Deng L and Yu D. 2013. Tensor deep stacking networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1944-1957[DOI:10.1109/tpami.2012.268]

Hwan Jin K, McCann M T, Froustey E and Unser M. 2017. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9):4509-4522[DOI:10.1109/TIP.2017.2713099]

Karoui M S, Deville Y, Benhalouche F Z and Boukerch I. 2017. Hypersharpening by joint-criterion nonnegative matrix factorization. IEEE Transactions on Geoscience and Remote Sensing, 55(3):1660-1670[DOI:10.1109/TGRS.2016.2628889]

Katkovnik V, Foi A, Egiazarian K and Astola J. 2010. From local kernel to nonlocal multiple-model image denoising. International Journal of Computer Vision, 86(1):1-32[DOI:10.1007/s11263-009-0272-7]

Li D R, Tong Q X, Li R X, Gong J Y and Zhang L P. 2012. Current issues in high-resolution Earth observation technology. Science China Earth Sciences, 42(6):805-813

李德仁, 童庆禧, 李荣兴, 龚健雅, 张良培. 2012.高分辨率对地观测的若干前沿科学问题.中国科学:地球科学, 42(6):805-813[DOI:10.1007/s11430-012-4445-9]

Li H, Liu F, Yang S Y and Zhang K. 2016. Remote sensing image fusion based on deep support value learning networks. Chinese Journal of Computers, 39(8):1583-1596

李红, 刘芳, 杨淑媛, 张凯. 2016.基于深度支撑值学习网络的遥感图像融合.计算机学报, 39(8):1583-1596[DOI:10.11897/SP.J.1016.2016.01583]

Liu P F, Xiao L and Li T. 2018. A variational pan-sharpening method based on spatial fractional-order geometry and spectral-spatial low-rank priors. IEEE Transactions on Geoscience and Remote Sensing, 56(3):1799-1802[DOI:10.1109/TGRS.2017.2768386]

Liu P F, Xiao L, Zhang J and Naz B. 2016. Spatial-hessian-feature-guided variational model for pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 54(4):2235-2253[DOI:10.1109/TGRS.2015.2497966]

Loncan L, de Almeida L B, Bioucas-Dias J M, Briottet X, Chanussot J, Dobigeon N, Fabre S, Liao W Z, Licciardi G A, Simões M, Tourneret J Y, Veganzones M A, Vivone G, Wei Q and Yokoya N. 2015. Hyperspectral pansharpening:a review. IEEE Geoscience and Remote Sensing Magazine, 3(3):27-46[DOI:10.1109/MGRS.2015.2440094]

Lu C Y, Tang J H, Yan S C and Lin Z C. 2014. Generalized nonconvex nonsmooth low-rank minimization//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE: 4130-4137[ DOI: 10.1109/CVPR.2014.526 http://dx.doi.org/10.1109/CVPR.2014.526 ]

Meinhardt T, Moeller M, Hazirbas C and Cremers D. 2017. Learning proximal operators: using denoising networks for regularizing inverse imaging problems//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 1799-1808[ DOI: 10.1109/ICCV.2017.198 http://dx.doi.org/10.1109/ICCV.2017.198 ]

Möller M, Wittman T, Bertozzi A L and Burger M. 2012. A variational approach for sharpening high dimensional images. SIAM Journal on Imaging Sciences, 5(1):150-178[DOI:10.1137/100810356]

Nezhad Z H, Karami A, Heylen R and Scheunders P. 2016. Fusion of hyperspectral and multispectral images using spectral unmixing and sparse coding. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6):2377-2389[DOI:10.1109/JSTARS.2016.2528339]

Nunez J, Otazu X, Fors O, Prades A, Pala V and Arbiol R. 1999. Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote Sensing, 37(3):1204-1211[DOI:10.1109/36.763274]

Palsson F, Sveinsson J R and Ulfarsson M O. 2017. Multispectral and hyperspectral image fusion using a 3-D-convolutional neural network. IEEE Geoscience and Remote Sensing Letters, 14(5):639-643[DOI:10.1109/lgrs.2017.2668299]

Qu Y, Qi H R and Kwan C. 2018. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE: 2511-2520[ DOI: 10.1109/CVPR.2018.00266 http://dx.doi.org/10.1109/CVPR.2018.00266 ]

Rick Chang J H, Li C L, Póczos B, Kumar B V K V and Sankaranarayanan A C. 2017. One network to solve them all——solving linear inverse problems using deep projection models//Proceedings of IEEE International Conference on Computer Vision. Venice: IEEE: 5889-5898[ DOI: 10.1109/ICCV.2017.627 http://dx.doi.org/10.1109/ICCV.2017.627 ]

Schowengerdt R A. 2007. Remote Sensing. Models and Methods for Image Processing. 3rd ed. The Netherlands: Elsevier

Shi W Z, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D and Wang Z H. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 1874-1883[ DOI: 10.1109/CVPR.2016.207 http://dx.doi.org/10.1109/CVPR.2016.207 ]

Sun J, Zhang Y, Wu Z B, Zhu Y Q, Yin X L, Ding Z Z, Wei Z H, Plaza J and Plaza A. 2019. An efficient and scalable framework for processing remotely sensed big data in cloud computing environments. IEEE Transactions on Geoscience and Remote Sensing, 57(7):4294-4308[DOI:10.1109/TGRS.2018.2890513]

Tong Q X, Xue Y Q and Zhang L F. 2014. Progress in hyperspectral remote sensing science and technology in china over the past three decades. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(1):70-91[DOI:10.1109/jstars.2013.2267204]

Tu T M, Su S C, Shyu H C and Huang P S. 2001. A new look at IHS-like image fusion methods. Information Fusion, 2(3):177-186[DOI:10.1016/s1566-2535(01)00036-7]

Veganzones M A, Simões M, Licciardi G, Yokoya N, Bioucas-Dias J M and Chanussot J. 2016. Hyperspectral super-resolution of locally low rank images from complementary multisource data. IEEE Transactions on Image Processing, 25(1):274-288[DOI:10.1109/TIP.2015.2496263]

Vivone G, Alparone L, Chanussot J, Mura M D, Garzelli A, Licciardi G A, Restaino R and Wald L. 2015. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5):2565-2586[DOI:10.1109/TGRS.2014.2361734]

Wang T T, Fang F, Li F and Zhang G X. 2019. High-quality Bayesian pansharpening. IEEE Transactions on Image Processing, 28(1):227-239[DOI:10.1109/TIP.2018.2866954]

Wang Z W, Liu D, Yang J C and Huang T S. 2015. Deep networks for image super-resolution with sparse prior//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago: IEEE: 370-378[ DOI: 10.1109/ICCV.2015.50 http://dx.doi.org/10.1109/ICCV.2015.50 ]

Wang Z Y, Yang Y Z, Wang Z W, Chang S Y, Han W, Yang J C and Huang T. 2015. Self-tuned deep super resolution//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston: IEEE: 1-8[ DOI: 10.1109/CVPRW.2015.7301266 http://dx.doi.org/10.1109/CVPRW.2015.7301266 ]

Wei Q, Bioucas-Dias J, Dobigeon N and Tourneret J Y. 2015a. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Transactions on Geoscience and Remote Sensing, 53(7):3658-3668[DOI:10.1109/TGRS.2014.2381272]

Wei Q, Dobigeon N and Tourneret J Y. 2015b. Bayesian fusion of multi-band images. IEEE Journal of Selected Topics in Signal Processing, 9(6):1117-1127[DOI:10.1109/JSTSP.2015.2407855]

Wei Y C, Yuan Q Q, Shen H F and Zhang L P. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10):1795-1799[DOI:10.1109/LGRS.2017.2736020]

Xie Q, Zhou M H, Zhao Q, Meng D Y, Zuo W M and Xu Z B. 2019. Multispectral and hyperspectral image fusion by MS/HS fusion net//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE: 1585-1594[ DOI: 10.1109/CVPR.2019.00168 http://dx.doi.org/10.1109/CVPR.2019.00168 ]

Xie T, Li S T, Fang L Y and Liu L C. 2019. Tensor completion via nonlocal low-rank regularization. IEEE Transactions on Cybernetics, 49(6):2344-2354[DOI:10.1109/TCYB.2018.2825598]

Xie Y, Qu Y Y, Tao D C, Wu W W and Zhang W S. 2016. Hyperspectral image restoration via iteratively regularized weighted schatten p -norm minimization. IEEE Transactions on Geoscience and Remote Sensing, 54(8):4642-4659[DOI:10.1109/TGRS.2016.2547879 ]

Xu Y, Wu Z B, Chanussot J and Wei Z H. 2019. Nonlocal patch tensor sparse representation for hyperspectral image super-resolution. IEEE Transactions on Image Processing, 28(6):3034-3047[DOI:10.1109/TIP.2019.2893530]

Xu Y, Wu Z B, Chanussot J, Comon P and Wei Z H. 2020. Nonlocal coupled tensor CP decomposition for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing, 58(1):348-362[DOI:10.1109/TGRS.2019.2936486]

Yang G, Yu S M, Dong H, Slabaugh G, Dragotti P L, Ye X J, Liu F D, Arridge S, Keegan J, Guo Y K and Firmin D. 2018. DAGAN:deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Transactions on Medical Imaging, 37(6):1310-1321[DOI:10.1109/TMI.2017.2785879]

Yang J F, Fu X Y, Hu Y W, Ding X H and Paisley J. 2017. PanNet: a deep network architecture for pan-sharpening//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 1753-1761[ DOI: 10.1109/ICCV.2017.193 http://dx.doi.org/10.1109/ICCV.2017.193 ]

Yokoya N, Grohnfeldt C and Chanussot J. 2017. Hyperspectral and multispectral data fusion:a comparative review of the recent literature. IEEE Geoscience and Remote Sensing Magazine, 5(2):29-56[DOI:10.1109/MGRS.2016.2637824]

Yuan Q Q, Wei Y C, Meng X C, Shen H F and Zhang L P. 2018. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3):978-989[DOI:10.1109/JSTARS.2018.2794888]

Zhang K, Wang M and Yang S Y. 2017a. Multispectral and hyperspectral image fusion based on group spectral embedding and low-rank factorization. IEEE Transactions on Geoscience and Remote Sensing, 55(3):1363-1371[DOI:10.1109/tgrs.2016.2623626]

Zhang K, Wang M, Yang S Y and Jiao L C. 2018. Spatial-spectral-graph-regularized low-rank tensor decomposition for multispectral and hyperspectral image fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(4):1030-1040[DOI:10.1109/JSTARS.2017.2785411]

Zhang K, Zuo W M, Gu S H and Zhang L. 2017b. Learning deep CNN denoiser prior for image restoration//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 2808-2817[ DOI: 10.1109/CVPR.2017.300 http://dx.doi.org/10.1109/CVPR.2017.300 ]

Zhang L P and Shen H F.2016. Progress and future of remote sensing data fusion. Journal of Remote Sensing, 20(5):1050-1061

张良培, 沈焕锋. 2016.遥感数据融合的进展与前瞻.遥感学报, 20(5):1050-1061[DOI:10.11834/jrs.20166243]

Zhang L P, Zhang L F and Du D. 2016. Deep learning for remote sensing data:a technical tutorial on the state of the art. IEEE Geoscience and Remote Sensing Magazine, 4(2):22-40[DOI:10.1109/MGRS.2016.2540798]

Zhu X X, Grohnfeldt C and Bamler R. 2016. Exploiting joint sparsity for pansharpening:the J-SparseFI algorithm. IEEE Transactions on Geoscience and Remote Sensing, 54(5):2664-2681[DOI:10.1109/TGRS.2015.2504261]

Zhu X X, Tuia D, Mou L C, Xia G S, Zhang L P and Xu F. 2017. Deep learning in remote sensing:a comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5(4):8-36[DOI:10.1109/MGRS.2017.2762307]