Hyperspectral and multispectral image fusion focused on error compensation

Shuai Fang; Man Xu

doi:10.11834/jig.220568

Remote Sensing Image Processing | Views : 0 下载量: 337 CSCD: 2

PDF
Export
Share
Collection
Album

Hyperspectral and multispectral image fusion focused on error compensation
Vol. 28, Issue 1, Pages: 277-289(2023)
Received：06 June 2022，

Revised：2022-10-10，

Accepted：17 October 2022，

Published：16 January 2023
DOI： 10.11834/jig.220568
稿件说明：

移动端阅览

DOI：

Shuai Fang, Man Xu. Hyperspectral and multispectral image fusion focused on error compensation[J]. Journal of Image and Graphics, 2023, 28(1): 277-289. DOI： 10.11834/jig.220568.

摘要

目的

在高光谱和多光谱融合领域，光谱字典学习是一种常用的方法，但是这种算法获得的融合结果不可避免地会损失一部分空间信息。针对这一问题，本文提出了一种面向误差补偿的高光谱和多光谱融合框架，即先利用低分辨率图像构建出误差，再将构建的误差对字典学习所得初步结果进行补偿，从而捕获更多的空间信息和光谱信息。

方法

设计了一种基于局部区域的注入系数，使得到的系数能够根据局部区域的光谱特性自适应地调整误差信息的注入权重，从而避免注入过多或者过少的空间信息而导致光谱失真。为了保证融合结果在提升空间分辨率的同时不发生光谱畸变，在梯度域提取多光谱图像的空间结构，构造变分模型，并将光谱字典学习得到的初步结果与计算出的系数相结合构成优化项并将该优化项与变分模型结合，构建出一个新的模型，通过迭代更新，逐步提升补偿系数的精度和融合结果的质量。

结果

分别在两个公开数据集上与其他算法进行对比，实验结果表明，本文算法在评价指标和视觉效果上都取得明显提升。从主观分析来看，本文方法可以得到融合质量高、视觉效果自然清晰的目标图像。从客观评价指标来看，在Pavia University数据集上的实验结果在ERGAS(relative dimensionless global error in synthesis)、SAM(spectral angle mapper)、RMSE(root mean square error)指标上与次优方法相比，分别提升了4.2%、4.1%和2.2%；在AVIRIS(airborne visible infrared imaging spectrometer)数据集上分别提升了2.0%，4.0%和3.5%。

结论

本文算法在有效提升融合结果空间分辨率的同时

很好地保持了光谱信息。且在不同数据集上取得了较优的表现，具有一定的鲁棒性。

Abstract

Objective

Hyperspectral images(HSI) are widely used in image classification and target detection because of their rich and useful spectral information. Their spatial resolution is required to be optimized further due to the limitation of imaging cameras. Multispectral or panchromatic images with lower spectral resolution have higher spatial resolution compared to hyperspectral images. To improve the spatial resolution of hyperspectral images

they are often fused with multispectral images(MSI) or panchromatic images in the same scene. Dictionary learning is one of the popular algorithms for image fusion

which can be divided into dictionary-spectral learning and dictionary-spatial learning. Using spectral dictionary learning algorithms

high-resolution hyper-spectral images can be expressed as an over-completed spectral dictionary multiplied by sparse coefficients. Spectral dictionary can illustrate the spectral information of high-resolution hyperspectral images

which is originated from k-singular value decomposition(SVD) and the algorithms-related. It is challenged of spatial information loss for spectral dictionary cannot fully express spatial information. Therefore

we develop a new compensated framework to explore more detailed spatial information. To improve the spatial resolution of fusion result

residual spatial information is used to compensate the preliminary result. The residual information is calculated as the error between the image obtained by spectral down sampling through preliminary results and the multispectral image. It is required to inject the residual space information into the preliminary results in an appropriate manner. However

most of the algorithms have limitations for that they only consider the differences between spectral channels but do not consider the differences in spatial. Therefore

the spectral and spatial quality of the fused image will be seriously affected if the extracted errors are injected indiscriminately into a channel.

Method

Our method is focused on improving the spatial resolution of hyperspectral image while keeping its spectral information free from distortion. On the basis of fully capturing the spectral information of hyperspectral image through dictionary learning

the residual spatial information is used to compensate for the spatial resolution. First

a fusion method based on local region is designed. To avoid the spectral distortion caused by inappropriate information injection

adjusting the injection degree of residual information adaptively through a coefficient in accordance with the spectral features of the local area. Second

to maintain the consistency of the spatial structure of the fusion results with multispectral image(MSI)

the spatial structure of MSI is extracted in the gradient domain

and a variational model is constructed. The coefficients-calculated are incorporated to the spectral dictionary-derived preliminary results to form an optimization term updated alternately. The objective function also contains spectral constraints composed of the target image and HSI

as well as spatial constraint composed of variational components. This function is run iteratively via alternating direction method of multipliers (ADMM). The spatial and spectral constraints-involved fusion model cannot only improve the spatial resolution

but also ensure that the spectral information does not have distortion.

Result

Our analysis is compared to six other algorithms on two public datasets. To verify the effectiveness and efficiency of our method

qualitative and quantitative evaluation is carried out in combination with other methods-related through the experimental platform—MATLAB R2018a. For qualitative analysis

our proposed method is capable to get the target image-fused with higher natural and clear visual effect. For quantitative evaluation indicators

compared with the sub-optimal experiment

the Pavia University data set-relevant experimental results are reduced by 4.2%

4.1% and 2.2% in relation to relative dimensionless global error in synhesis(ERGAS)

spectral angle mapper(SAM) and root mean square error(RMSE) indicators. The AVIRIS(airborne visible infrared imaging spectrometer) dataset-related values are reduced by 2.0%

4.0% and 3.5% of each.

Conclusion

Our fusion algorithm can effectively improve the spatial resolution preserve spectral information. Furthermore

our algorithm has its optimization and robustness potentially.

关键词

Keywords

references

Akhtar N, Shafait F and Mian A. 2014. Sparse spatio-spectral representation for hyperspectral image super-resolution//Proceedings of the 13th European Conference Computer Vision. Zurich, Switzerland: Springer: 63-78[ DOI: 10.1007/978-3-319-10584-0_5 http://dx.doi.org/10.1007/978-3-319-10584-0_5 ]

Akhtar N, Shafait F and Mian A. 2015. Bayesian sparse representation for hyperspectral image super resolution//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE: 3631-3640[ DOI: 10.1109/CVPR.2015.7298986 http://dx.doi.org/10.1109/CVPR.2015.7298986 ]

Alparone L, Wald L, Chanussot J, Thomas C, Gamba P and Bruce L M. 2007. Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10): 3012-3021[DOI: 10.1109/TGRS.2007.904923]

Boyd S, Parikh N, Chu E, Peleato B and Eckstein J. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends Ⓡ in Machine Learning, 3(1): 1-122[DOI: 10.1561/2200000016 ]

Dian R W, Li S T, Fang L Y and Bioucas-Dias J. 2018a. Hyperspectral image super-resolution via local low-rank and sparse representations//IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia, Spain: IEEE: 4003-4006[ DOI: 10.1109/IGARSS.2018.8519213 http://dx.doi.org/10.1109/IGARSS.2018.8519213 ]

Dian R W, Li S T, Fang L Y and Wei Q. 2019. Multispectral and hyperspectral image fusion with spatial-spectral sparse representation. Information Fusion, 49: 262-270[DOI: 10.1016/j.inffus.2018.11.012]

Dian R W, Li S T, Guo A J and Fang L Y. 2018b. Deep hyperspectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems, 29(11): 5345-5355[DOI: 10.1109/TNNLS.2018.2798162]

Dong C, Loy C C, He K M and Tang X O. 2014. Learning a deep convolutional network for image super-resolution//Proceedings of the 13th European Conference Computer Vision. Zurich, Switzerland: Springer: 184-199[ DOI: 10.1007/978-3-319-10593-2_13 http://dx.doi.org/10.1007/978-3-319-10593-2_13 ]

Dong W S, Fu F Z, Shi G M, Cao X, Wu J J, Li G Y and Li X. 2016. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing, 25(5): 2337-2352[DOI: 10.1109/TIP.2016.2542360]

Fang L Y, Zhuo H J and Li S T. 2018. Super-resolution of hyperspectral image via superpixel-based sparse representation. Neurocomputing, 273: 171-177[DOI: 10.1016/j.neucom.2017.08.019]

Fang S, Chao L and Cao F Y. 2020. New pan-sharpening method based on adaptive weight mechanism. Journal of Image and Graphics, 25(3): 546-557

方帅, 潮蕾, 曹风云. 2020. 自适应权重注入机制遥感图像融合. 中国图象图形学报, 25(3): 546-557[DOI: 10.11834/jig.190280]

Han X H, Shi B X and Zheng Y Q. 2018. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Transactions on Image Processing, 27(11): 5625-5637[DOI: 10.1109/TIP.2018.2855418]

Han X L, Yu J, Xue J H and Sun W D. 2020. Hyperspectral and multispectral image fusion using optimized twin dictionaries. IEEE Transactions on Image Processing, 29: 4709-4720[DOI: 10.1109/TIP.2020.2968773]

Hardie R C, Eismann M T and Wilson G L. 2004. MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor. IEEE Transactions on Image Processing, 13(9): 1174-1184[DOI: 10.1109/TIP.2004.829779]

Jiao J and Wu L D. 2019. Fusion of multispectral and panchromatic images via morphological filter and improved PCNN in NSST domain. Journal of Image and Graphics, 24(3): 435-446

焦姣, 吴玲达. 2019. 形态学滤波和改进PCNN的NSST域多光谱与全色图像融合. 中国图象图形学报, 24(3): 435-446[DOI: 10.11834/jig.180399]

Kawakami R, Matsushita Y, Wright J, Ben-Ezra M, Tai Y W and Ikeuchi K. 2011. High-resolution hyperspectral imaging via matrix factorization//CVPR 2011. Colorado Springs, USA: IEEE: 2329-2336[ DOI: 10.1109/CVPR.2011.5995457 http://dx.doi.org/10.1109/CVPR.2011.5995457 ]

Simões M, Bioucas-Dias J, Almeida L B and Chanussot J. 2015. A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Transactions on Geoscience and Remote Sensing, 53(6): 3373-3388[DOI: 10.1109/TGRS.2014.2375320]

Wang X Y, Zhong Y F, Zhang L P and Xu Y Y. 2017. Spatial group sparsity regularized nonnegative matrix factorization for hyperspectral unmixing. IEEE Transactions on Geoscience and Remote Sensing, 55(11): 6287-6304[DOI: 10.1109/TGRS.2017.2724944]

Wei Q, Dobigeon N and Tourneret J Y. 2015. Fast fusion of multi-band images based on solving a Sylvester equation. IEEE Transactions on Image Processing, 24(11): 4109-4121[DOI: 10.1109/TIP.2015.2458572]

Xie Q, Zhou M H, Zhao Q, Xu Z B and Meng D Y. 2022. MHF-Net: an interpretable deep network for multispectral and hyperspectral image fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3): 1457-1473[DOI: 10.1109/TPAMI.2020.3015691]

Xu T, Huang T Z, Deng L J and Yokoya N. 2022. An iterative regularization method based on tensor subspace representation for hyperspectral image super-resolution. IEEE Transactions on Geoscience and Remote Sensing, 60: #5529316[DOI: 10.1109/TGRS.2022.3176266]

Yang J X, Zhao Y Q and Chan J C W. 2018. Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network. Remote Sensing, 10(5): #800[DOI: 10.3390/rs10050800]

Yang Y, Wu L, Huang S Y, Wan W G, Tu W and Lu H Y. 2020. Multiband remote sensing image pansharpening based on dual-injection model. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13: 1888-1904[DOI: 10.1109/JSTARS.2020.2981975]

Yin H T and Li S T. 2015. Pansharpening with multiscale normalized nonlocal means filter: a two-step approach. IEEE Transactions on Geoscience and Remote Sensing, 53(10): 5734-5745[DOI: 10.1109/TGRS.2015.2429691]

Yokoya N, Yairi T and Iwasaki A. 2012. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscienceand Remote Sensing, 50(2): 528-537[DOI:10.1109/TGRS.2011.2161320]

Alert me when the article has been cited

提交

Semantic segmentation benchmark dataset for coastal ecosystem monitoring based on unmanned aerial vehicle （UAV）

Multimodal remote sensing image registration： a survey

Image style transfer via curved stroke rendering

High-order conditional random fields-relevant multi-branch polarimetric SAR image classification

Hyperspectral image classification with enhanced class separability