浏览全部资源
扫码关注微信
收稿：2022-11-04，
修回：2023-03-20，
纸质出版：2023-12-16
移动端阅览
目的
2
将高光谱图像和多光谱图像进行融合，可以获得具有高空间分辨率和高光谱分辨率的光谱图像，提升光谱图像的质量。现有的基于深度学习的融合方法虽然表现良好，但缺乏对多源图像特征中光谱和空间长距离依赖关系的联合探索。为有效利用图像的光谱相关性和空间相似性，提出一种联合自注意力的Transformer网络来实现多光谱和高光谱图像融合超分辨。
方法
2
首先利用联合自注意力模块，通过光谱注意力机制提取高光谱图像的光谱相关性特征，通过空间注意力机制提取多光谱图像的空间相似性特征，将获得的联合相似性特征用于指导高光谱图像和多光谱图像的融合；随后，将得到的融合特征输入到基于滑动窗口的残差Transformer深度网络中，探索融合特征的长距离依赖信息，学习深度先验融合知识；最后，特征通过卷积层映射为高空间分辨率的高光谱图像。
结果
2
在CAVE和Harvard光谱数据集上分别进行了不同采样倍率下的实验，实验结果表明，与对比方法相比，本文方法从定量指标和视觉效果上，都取得了更好的效果。本文方法相较于性能第二的方法EDBIN（enhanced deep blind iterative network），在CAVE数据集上峰值信噪比提高了0.5 dB，在Harvard数据集上峰值信噪比提高了0.6 dB。
结论
2
本文方法能够更好地融合光谱信息和空间信息，显著提升高光谱融合超分图像的质量。
Objective
2
Hyperspectral image （HSI） contains rich spectral information and has advantages over multispectral image （MSI） in accurately distinguishing different types of materials. Therefore， HSI has been widely used in many computer vision tasks， including vegetation detection， face recognition， and feature segmentation. However， due to the limitations in hardware equipment and the acquisition environment， an inevitable trade-off arises between spatial resolution and spectral resolution. Thus， HSIs under real scenes often have low spatial resolution， which negatively affects the performance of subsequent vision tasks. By fusing the low-resolution HSI （LR-HSI） with a high-resolution MSI （HR-MSI） under the same scene using the HSI super-resolution algorithm， the spatial resolution of HSIs can be effectively improved. Existing HSI fusion algorithms can be roughly classified into traditional-model-based and deep-learning-based methods. Traditional-model-based fusion methods employ various handcrafted shallow priors （e.g
.
， matrix/tensor factorization， total variation， and low rank） to utilize the intrinsic statistics of observed spectral images. However， these methods lack generalization ability to complex real scenarios and consume much time in iteratively optimizing the designed prior. Meanwhile， deep-learning-based fusion methods can automatically learn the prior knowledge from large-scale datasets. Although these methods often achieve better fusion results compared with traditional-model-based fusion methods， they do not jointly explore the inner self-similarity of multi-source spectral images， where the LR-HSI shows high correlation in the spectral dimension and the HR-MSI shows spatial similarities in texture and edges. In addition， the weights of these convolution-based networks are learned during training but are fixed during testing， hence limiting the potential adaptability of networks. To effectively exploit the inner spatial and spectral similarity of spectral images， we propose an MSI and HSI fusion network with a joint self-attention fusion module and Transformer.
Method
2
Given that LR-HSI has reliable information in the spectral dimension， the critical task of the HSI fusion method is to fill the missing texture details in the spatial dimension without losing discriminable spectral information. Given the LR-HSI and its matching HR-MSI， our proposed method fuses these two spectral images to obtain the desired HR-HSI in three steps. First， the similarity information of LR-HSI and HR-MSI is extracted by the joint self-attention module. Specifically， the spectral similarity features from LR-HSI are extracted by the channel attention module， and the spatial similarity features from HR-MSI are extracted by the spatial attention module. The obtained similarity features are then used to guide the fusion process. Second， to achieve a deep representation and explore the long-range dependencies of the fusion features， the preliminary fusion features are fed into the deep Transformer network， which comprises a shift window attention module， LayerNorm， and multilayer perceptron. The convolution layer and skip connection are also included in the proposed Transformer fusion network to further enhance the model flexibility. Third， the fusion features from Transformer are mapped to the desired high-resolution HSI. The overall network is implemented by the Pytorch framework and trained in an end-to-end manner. To generate training data， the training images are cropped to the size of 96 × 96 × 31， resulting in approximately 8 000 training patches that are smoothed by a Gaussian blur kernel and spatially down-sampled to obtain LR-HSI. The MSI images are generated by the spectral response function of a Nikon D700 camera.
Result
2
We compare our method with seven state-of-the-art fusion methods， including one traditional-model-based method and six deep-learning-based methods. The peak-signal-to-noise ratio （PSNR）， structural similarity index measure （SSIM）， erreur relative globale adimensionnelle de Synthèse （ERGAS）， and spectral angle mapper （SAM） are utilized as quantitative metrics in evaluating the performance of these fusion methods. To verify the effectiveness of the proposed model， we perform experiments on two widely used HSI datasets， namely， the CAVE and Harvard datasets. For the CAVE dataset， the first 20 images are selected for training， and the last 12 images are used for testing. Similarly， for the Harvard dataset， the first 30 images are selected for training， and the last 20 images are used for testing. Experimental results under different scale factors show that the proposed method achieves better fusion results in terms of quantitative metrics and visual effects compared to the other state-of-the-art methods. Under a scale factor of 8， the PSNR， SAM， and ERGAS of the proposed method is improved by 0.5 dB， 0.13， and 0.2， respectively， compared to EDBIN， which is the second best-performing method on the CAVE dataset. Under a scale factor 16， the PSNR of the proposed method is improved by at least 0.4 dB compared to the other methods on the Harvard dataset. The visual results show that our proposed method outperforms the other methods in recovering both fine-grained spatial textures and spectral details. The ablation study also proves that the employed Transformer fusion network significantly improves the fusion process.
Conclusion
2
In this paper， we propose a Transformer-based MSI and HSI fusion network with a joint self-attention fusion module， which can effectively utilize the spectral similarity of LR-HSI and the spatial similarity of HR-MSI to guide the fusion process through a 2D attention mechanism. The preliminary fusion results pass through the residual Transformer network to obtain a deep feature representation and to reconstruct the desired HR-HSI. Qualitative and quantitative experiments show that the proposed method has better spectral fidelity and spatial resolution compared to the state-of-the-art HSI fusion methods.
Adão T ， Hruška J ， Pádua L ， Bessa J ， Peres E ， Morais R and Sousa J J . 2017 . Hyperspectral imaging： a review on UAV-based sensors， data processing and applications for agriculture and forestry . Remote Sensing ， 9 （ 11 ）： # 1110 ［ DOI： 10.3390/rs9111110 http://dx.doi.org/10.3390/rs9111110 ］
Akhtar N ， Shafait F and Mian A . 2015 . Bayesian sparse representation for hyperspectral image super resolution // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston， USA ： IEEE： 3631 - 3640 ［ DOI： 10.1109/CVPR.2015.7298986 http://dx.doi.org/10.1109/CVPR.2015.7298986 ］
Chen D J ， Hsieh H Y and Liu T L . 2021 . Adaptive image Transformer for one-shot object detection // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville， USA ： IEEE： 12242 - 12251 ［ DOI： 10.1109/CVPR46437.2021.01207 http://dx.doi.org/10.1109/CVPR46437.2021.01207 ］
Dong W S ， Fu F Z ， Shi G M ， Cao X ， Wu J J ， Li G Y and Li X . 2016 . Hyperspectral image super-resolution via non-negative structured sparse representation . IEEE Transactions on Image Processing ， 25 （ 5 ）： 2337 - 2352 ［ DOI： 10.1109/TIP.2016.2542360 http://dx.doi.org/10.1109/TIP.2016.2542360 ］
Dong W S ， Zhou C ， Wu F F ， Wu J J ， Shi G M and Li X . 2021 . Model-Guided deep hyperspectral image super-resolution . IEEE Transactions on Image Processing ， 30 ： 5754 - 5768 ［ DOI： 10.1109/TIP.2021.3078058 http://dx.doi.org/10.1109/TIP.2021.3078058 ］
Dosovitskiy A ， Beyer L ， Kolesnikov A ， Weissenborn D ， Zhai X H ， Unterthiner T ， Dehghani M ， Minderer M ， Heigold G ， Gelly S ， Uszkoreit J and Houlsby N . 2021 . An image is worth 16 × 16 words： Transformers for image recognition at scale ［EB/OL］. ［ 2023-09-10 ］. http://arxiv.org/pdf/2010.11929.pdf http://arxiv.org/pdf/2010.11929.pdf
Han X H ， Shi B X and Zheng Y Q . 2018 . Self-similarity constrained sparse representation for hyperspectral image super-resolution . IEEE Transactions on Image Processing ， 27 （ 11 ）： 5625 - 5637 ［ DOI： 10.1109/TIP.2018.2855418 http://dx.doi.org/10.1109/TIP.2018.2855418 ］
Hu J F ， Huang T Z ， Deng L J ， Jiang T X ， Vivone G and Chanussot J . 2021 . Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks . IEEE Transactions on Neural Networks and Learning Systems ， 33 （ 12 ）： 7251 - 7265 ［ DOI： 10.1109/TNNLS.2021.3084682 http://dx.doi.org/10.1109/TNNLS.2021.3084682 ］
Kawakami R ， Matsushita Y ， Wright J ， Ben-Ezra M ， Tai Y W and Ikeuchi K . 2011 . High-resolution hyperspectral imaging via matrix factorization // Proceedings of the CVPR 2011 . Colorado Springs， USA ： IEEE： 2329 - 2336 ［ DOI： 10.1109/CVPR.2011.5995457 http://dx.doi.org/10.1109/CVPR.2011.5995457 ］
Kingma D P and Ba J . 2017 . Adam： a method for stochastic optimization ［EB/OL］. ［ 2022-05-25 ］. http://arxiv.org/pdf/1412.6980.pdf http://arxiv.org/pdf/1412.6980.pdf
Li K ， Dai D X and Van Gool L . 2022 . Hyperspectral image super-resolution with RGB image super-resolution as an auxiliary task // Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision . Waikoloa， USA ： IEEE： 4039 - 4048 ［ DOI： 10.1109/WACV51458.2022.00409 http://dx.doi.org/10.1109/WACV51458.2022.00409 ］
Li W ， Du Q and Zhang B . 2015 . Combined sparse and collaborative representation for hyperspectral target detection . Pattern Recognition ， 48 （ 12 ）： 3904 - 3916 ［ DOI： 10.1016/j.patcog.2015.05.024 http://dx.doi.org/10.1016/j.patcog.2015.05.024 ］
Li X S ， Zhang Y Q ， Ge Z X ， Cao G ， Shi H and Fu P . 2021 . Adaptive nonnegative sparse representation for hyperspectral image super-resolution . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing ， 14 ： 4267 - 4283 ［ DOI： 10.1109/JSTARS.2021.3072044 http://dx.doi.org/10.1109/JSTARS.2021.3072044 ］
Liang J ， Zhou J ， Bai X and Qian Y T . 2013 . Salient object detection in hyperspectral imagery // Proceedings of 2013 IEEE International Conference on Image Processing . Melbourne， Australia ： IEEE： 2393 - 2397 ［ DOI： 10.1109/ICIP.2013.6738493 http://dx.doi.org/10.1109/ICIP.2013.6738493 ］
Liang J Y ， Cao J Z ， Sun G L ， Zhang K ， Van Gool L and Timofte R . 2021 . SwinIR： image restoration using Swin Transformer // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops . Montreal， Canada ： IEEE： 1833 - 1844 ［ DOI： 10.1109/ICCVW54120.2021.00210 http://dx.doi.org/10.1109/ICCVW54120.2021.00210 ］
Liu J J ， Wu Z B ， Xiao L ， Sun J and Yan H . 2020 . A truncated matrix decomposition for hyperspectral image super-resolution . IEEE Transactions on Image Processing ， 29 ： 8028 - 8042 ［ DOI： 10.1109/TIP.2020.3009830 http://dx.doi.org/10.1109/TIP.2020.3009830 ］
Liu Z ， Lin Y T ， Cao Y ， Hu H ， Wei Y X ， Zhang Z ， Lin S and Guo B N . 2021 . Swin Transformer： hierarchical vision Transformer using shifted windows // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision . Montreal， Canada ： IEEE： 9992 - 10002 ［ DOI： 10.1109/ICCV48922.2021.00986 http://dx.doi.org/10.1109/ICCV48922.2021.00986 ］
Peng Y D ， Li W S ， Luo X B and Du J . 2021 . Hyperspectral image superresolution using global gradient sparse and nonlocal low-rank tensor decomposition with hyper-laplacian prior . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing ， 14 ： 5453 - 5469 ［ DOI： 10.1109/JSTARS.2021.3076170 http://dx.doi.org/10.1109/JSTARS.2021.3076170 ］
Ronneberger O ， Fischer P and Brox T . 2015 . U-Net： convolutional networks for biomedical image segmentation // Proceedings of 2021 IEEE/CVF International Conference on Computer the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention . Munich， Germany ： Springer： 234 - 241 ［ DOI： 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ］
Simões M ， Bioucas-Dias J ， Almeida L B and Chanussot J . 2015 . A convex formulation for hyperspectral image superresolution via subspace-based regularization . IEEE Transactions on Geoscience and Remote Sensing ， 53 （ 6 ）： 3373 - 3388 ［ DOI： 10.1109/TGRS.2014.2375320 http://dx.doi.org/10.1109/TGRS.2014.2375320 ］
Uzair M ， Mahmood A and Mian A . 2015 . Hyperspectral face recognition with spatiospectral information fusion and PLS regression . IEEE Transactions on Image Processing ， 24 （ 3 ）： 1127 - 1137 ［ DOI： 10.1109/TIP.2015.2393057 http://dx.doi.org/10.1109/TIP.2015.2393057 ］
Vaswani A ， Shazeer N ， Parmar N ， Uszkoreit J ， Jones L ， Gomez A N ， Kaiser Ł and Polosukhin I . 2017 . Attention is all you need // Proceedings of the 31st International Conference on Neural Information Processing Systems . Long Beach， USA ： Curran Associates Inc.： 6000 - 6010
Wang W ， Fu X Y ， Zeng W H ， Sun L Y ， Zhan R H ， Huang Y and Ding X H . 2021a . Enhanced deep blind hyperspectral image fusion . IEEE Transactions on Neural Networks and Learning Systems ， 34 （ 3 ）： 1513 - 1523 ［ DOI： 10.1109/TNNLS.2021.3105543 http://dx.doi.org/10.1109/TNNLS.2021.3105543 ］
Wang W H ， Xie E Z ， Li X ， Fan D P ， Song K T ， Liang D ， Lu T ， Luo P and Shao L . 2021b . Pyramid vision Transformer： a versatile backbone for dense prediction without convolutions // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision . Montreal， Canada ： IEEE： 548 - 558 ［ DOI： 10.1109/ICCV48922.2021.00061 http://dx.doi.org/10.1109/ICCV48922.2021.00061 ］
Wang Y ， Chen X ， Han Z and He S Y . 2017 . Hyperspectral image super-resolution via nonlocal low-rank tensor approximation and total variation regularization . Remote Sensing ， 9 （ 12 ）： # 1286 ［ DOI： 10.3390/rs9121286 http://dx.doi.org/10.3390/rs9121286 ］
Wang Z D ， Cun X ， Bao J M ， Zhou W G ， Liu J Z and Li H Q . 2022 . Uformer： a general U-shaped Transformer for image restoration // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 17662 - 17672 ［ DOI： 10.1109/CVPR52688.2022.01716 http://dx.doi.org/10.1109/CVPR52688.2022.01716 ］
Xie Q ， Zhou M H ， Zhao Q ， Xu Z B and Meng D Y . 2022 . MHF-Net： an interpretable deep network for multispectral and hyperspectral image fusion . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 44 （ 3 ）， 1457 - 1473 ［ DOI： 10.1109/TPAMI.2020.3015691 http://dx.doi.org/10.1109/TPAMI.2020.3015691 ］
Yao J ， Hong D F ， Chanussot J ， Meng D Y ， Zhu X X and Xu Z B . 2020 . Cross-Attention in coupled unmixing nets for unsupervised hyperspectral super-resolution // Proceedings of 2021 IEEE/CVF International Conference on Computer 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 208 - 224 ［ DOI： 10.1007/978-3-030-58526-6_13 http://dx.doi.org/10.1007/978-3-030-58526-6_13 ］
Zamir S W ， Arora A ， Khan S ， Hayat M ， Khan F S and Yang M H . 2022 . Restormer： efficient Transformer for high-resolution image restoration // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 5718 - 5729 ［ DOI： 10.1109/CVPR52688.2022.00564 http://dx.doi.org/10.1109/CVPR52688.2022.00564 ］
Zhang K ， Wang M ， Yang S Y and Jiao L C . 2018a . Spatial-Spectral-Graph-Regularized low-rank tensor decomposition for multispectral and hyperspectral image fusion . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing ， 11 （ 4 ）： 1030 - 1040 ［ DOI： 10.1109/JSTARS.2017.2785411 http://dx.doi.org/10.1109/JSTARS.2017.2785411 ］
Zhang L ， Nie J T ， Wei W ， Zhang Y N ， Liao S C and Shao L . 2020 . Unsupervised adaptation learning for hyperspectral imagery super-resolution // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 3070 - 3079 ［ DOI： 10.1109/CVPR42600.2020.00314 http://dx.doi.org/10.1109/CVPR42600.2020.00314 ］
Zhang L ， Wei W ， Bai C C ， Gao Y F and Zhang Y N . 2018b . Exploiting clustering manifold structure for hyperspectral imagery super-resolution . IEEE Transactions on Image Processing ， 27 （ 12 ）： 5969 - 5982 ［ DOI： 10.1109/TIP.2018.2862629 http://dx.doi.org/10.1109/TIP.2018.2862629 ］
Zhang L ， Nie J T ， Wei W ， Li Y and Zhang Y N . 2021 . Deep blind hyperspectral image super-resolution . IEEE Transactions on Neural Networks and Learning Systems ， 32 （ 6 ）： 2388 - 2400 Li［ DOI： 10.1109/TNNLS.2020.3005234 http://dx.doi.org/10.1109/TNNLS.2020.3005234 ］
Zhao C Q ， Wang H H ， Zhao J J ， Ji L W ， Wang Q D ， Li H Z and Zhao Z J . 2022 . Cerebral stroke detection algorithm for visual Transformer and multi-feature fusion . Journal of Image and Graphics ， 27 （ 3 ）： 923 - 934
赵琛琦 ， 王华虎 ， 赵涓涓 ， 冀伦文 ， 王麒达 ， 李慧芝 ， 赵紫娟 . 2022 . 视觉Transformer与多特征融合的脑卒中检测算法 . 中国图象图形学报 ， 27 （ 3 ）： 923 - 934 ［ DOI： 10.11834/jig.210745 http://dx.doi.org/10.11834/jig.210745 ］
相关作者
相关机构
京公网安备11010802024621
