浏览全部资源
扫码关注微信
Published： 16 January 2023 ，
Accepted： 25 August 2022
移动端阅览
Xuhui Sun, Zheng Guan, Xue Wang. Vision transformer for fusing infrared and visible images in groups. [J]. Journal of Image and Graphics 28(1):166-178(2023)
目的
2
针对传统红外与可见光图像融合方法中人工设计特征提取和特征融合的局限性，以及基于卷积神经网络(convolutional neural networks，CNN)的方法无法有效提取图像中的全局上下文信息和特征融合过程中融合不充分的问题，本文提出了基于视觉Transformer和分组渐进式融合策略的端到端无监督图像融合网络。
方法
2
首先，将在通道维度上进行自注意力计算的多头转置注意力模块和通道注意力模块组合成视觉Transformer，多头转置注意力模块解决了自注意力计算量随像素大小呈次方增大的问题，通道注意力可以强化突出特征。其次，将CNN和设计的视觉Transformer并联组成局部—全局特征提取模块，用来提取源图像中的局部细节信息和全局上下文信息，使提取的特征既具有通用性又具有全局性。此外，为了避免融合过程中信息丢失，通过将特征分组和构造渐进残差结构的方式进行特征融合。最后，通过解码融合特征得到最终的融合图像。
结果
2
实验在TNO数据集和RoadScene数据集上与6种方法进行比较。主观上看，本文方法能够有效融合红外图像和可见光图像中的互补信息，得到优质的融合图像。从客观定量分析来说，在TNO数据集中，本文方法相比于性能第2的方法，在归一化互信息、非线性相关信息熵、平均梯度和标准差指标上分别提高了30.90%、0.58%、11.72%和11.82%；在RoadScene数据集中，本文方法在以上4个指标上依然取得了最好成绩。
结论
2
由于有效特征的提取和融合的复杂性以及融合过程中噪声的干扰，现有的融合方法都存在一定的局限性，或者融合效果质量不理想。相比之下，本文提出的基于视觉Transformer的图像融合方法在融合质量上取得了巨大提升，能够有效突出红外显著目标，并且将源图像中的背景信息和细节纹理有效保留在融合图像中，在对比度和清晰度方面也具有优越性。
Objective
2
Current image fusion can be as one the key branches of information fusion. Infrared and visible image fusion (IVF) is developed for image fusion dramatically. A visible light sensor-derived image can capture light-reflected. It is rich in texture detail information and fit to the human eye observation pattern. The image-fused can integrate rich detail information and thermal radiation information. Therefore
it is essential for such applications like object tracking
video surveillance
and autonomous driving. To resolve the constraints of manual-designed feature extraction and feature fusion in traditional infrared and visible image fusion methods
as well as the problems that convolutional neural network based (CNN-based) methods cannot effectively extract global contextual information in images and inadequate fusion during feature fusion
We develop an visual transformer-based end-to-end unsupervised fusion network via group-layered fusion strategies.
Method
2
First
a channel attention-based transformer is designed
which enhances the features further through computing the self-attention in the channel dimension and using the channel attention in series as the feed-forward network of the transformer. After that
to extract features from the source image
the transformer module and CNN are used in parallel to form a local-global feature extraction module. The features-extracted have the generality of the features extracted by the CNN model to avoid manual design of extraction rules. The global nature of the features extracted by the transformer can be used to make up for the shortage of convolutional operations. In addition
to alleviate the loss of feature information
we design a newly layer-grouped fusion module to fuse the local-global features-extracted by grouping the features of multiple sources in the channel dimension
fusing the features of the corresponding groups initially
and then fusing the features of different groups via a hierarchical residual structure.
Result
2
Our analysis is experimented on publicly available datasets TNO and RoadScene in comparison with six popular methods
which include traditional and deep learning-based methods both. Qualitative and quantitative evaluation methods are used to assess its effectiveness together. The qualitative analysis focuses on the clarity
contrast of the images perceived by the human eye. On the basis of qualitative evaluation
our method is capable to res
tore information-added in the infrared and visible images more effectively and maximize the useful information. At the same time
the fused images have contrast and definition and visual effects better. A quantitative-based comparison is carried out using six different metrics as well. On the TNO dataset
the proposed method achieved the best results in metrics normalized mutual information (NMI)
nonlinear correlation information entropy (
$$Q_{\mathrm{NICE}}$$
)
average gradient (AG) and standard deviation (SD)
improving by 30.90%
0.58%
11.72% and 11.82% compared to the second method. On the RoadScene dataset
the method achieves the best results in metrics normalized mutual information (NMI)
nonlinear correlation information entropy (
$$Q_{\mathrm{NICE}}$$
)
average gradient (AG)
standard deviation (SD) and visual fidelity improving by 32.74%
0.64%
24.53%
31.40%
31.73% compared to the second method.
Conclusion
2
Due to the complexity of effective feature extraction and fusion as well as the interference of noise in the fusion process
existing fusion methods have some challenging issues on fusion quality. In contrast
the visual transformer-based method has its potentials: 1) the infrared salient targets are highlighted effectively
2) the background information and detailed textures in the source image are retained in related to image fusion
and 3) the contrast and definition are optimized as well. The future research can be concerned of designing more general and efficient image fusion algorithms beyond the fusion of infrared and visible images.
红外图像可见光图像图像融合视觉Transformer卷积神经网络(CNN)
infrared imagevisible imageimage fusionvisual transformerconvolutional neural network (CNN)
Aslantas V and Bendes E. 2015. A new image quality metric for image fusion: the sum of the correlations of differences. AEU-International Journal of Electronics and Communications, 69(12): 1890-1896 [DOI: 10.1016/j.aeue.2015.09.004]
Cui G M, Feng H J, Xu Z H, Li Q and Chen Y T. 2015. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition. Optics Communications, 341: 199-209 [DOI: 10.1016/j.optcom.2014.12.032]
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X H, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J and Houlsby N. 2021. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2022-05-20].https://arxiv.org/pdf/2010.11929.pdfhttps://arxiv.org/pdf/2010.11929.pdf
Fu Y and Wu X J. 2021. A dual-branch network for infrared and visible image fusion//Proceedings of the 25th International Conference on Pattern Recognition. Milan, Italy: IEEE: 10675-10680[DOI: 10.1109/ICPR48806.2021.9412293http://dx.doi.org/10.1109/ICPR48806.2021.9412293]
Gao S H, Cheng M M, Zhao K, Zhang X Y, Yang M H and Torr P. 2021. Res2 Net: a new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2): 652-662 [DOI: 10.1109/TPAMI.2019.2938758]
Gong R and Wang X C. 2019. Infrared and visible image fusion based on BEMD and W-transform. Journal of Image and Graphics, 24(6): 987-999
宫睿, 王小春. 2019. BEMD分解和W变换相结合的红外与可见光图像融合. 中国图象图形学报, 24(6): 987-999 [DOI: 10.11834/jig.180530]
Han Y, Cai Y Z, Cao Y and Xu X M. 2013. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14(2): 127-135 [DOI: 10.1016/j.inffus.2011.08.002]
Hu J, Shen L, Albanie S, Sun G and Wu E H. 2020. Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8): 2011-2023 [DOI: 10.1109/TPAMI.2019.2913372]
Huang G, Liu Z, Van Der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2261-2269[DOI: 10.1109/CVPR.2017.243http://dx.doi.org/10.1109/CVPR.2017.243]
Huo X, Zou Y, Chen Y and Tan J Q. 2021. Dual-scale decomposition and saliency analysis based infrared and visible image fusion. Journal of Image and Graphics, 26(12): 2813-2825
霍星, 邹韵, 陈影, 檀结庆. 2021. 双尺度分解和显著性分析相结合的红外与可见光图像融合. 中国图象图形学报, 26(12): 2813-2825 [DOI: 10.11834/jig.200405]
Johnson J, Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 694-711[DOI: 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43]
Li H and Wu X J. 2019. DenseFuse: a fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5): 2614-2623 [DOI: 10.1109/TIP.2018.2887342]
Li H, Wu X J and Kittler J. 2018. Infrared and visible image fusion using a deep learning framework//Proceedings of the 24th International Conference on Pattern Recognition. Beijing, China: IEEE: 2705-2710[DOI: 10.1109/ICPR.2018.8546006http://dx.doi.org/10.1109/ICPR.2018.8546006]
Li H, Wu X J and Kittler J. 2020. MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Transactions on Image Processing, 29: 4733-4746 [DOI: 10.1109/TIP.2020.2975984]
Li H, Wu X J and Kittler J. 2021. RFN-Nest: an end-to-end residual fusion network for infrared and visible images. Information Fusion, 73: 72-86 [DOI: 10.1016/j.inffus.2021.02.023]
Li J, Zhu J M, Li C, Chen X and Yang B. 2022. CGTF: convolution-guided transformer for infrared and visible image fusion. IEEE Transactions on Instrumentation and Measurement, 71: #5012314 [DOI: 10.1109/TIM.2022.3175055]
Li S T, Yin H T and Fang L Y. 2012. Group-sparse representation with dictionary learning for medical image denoising and fusion. IEEE Transactions on Biomedical Engineering, 59(12): 3450-3459 [DOI: 10.1109/TBME.2012.2217493]
Liu Z, Blasch E, Xue Z Y, Zhao J Y, Laganiere R and Wu W. 2012. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(1): 94-109 [DOI: 10.1109/TPAMI.2011.109]
Liu Z, Lin Y T, Cao Y, Hu H, Wei Y X, Zhang Z, Lin S and Guo B N. 2021. Swin transformer: hierarchical vision transformer using shifted windows//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE: 9992-10002[DOI: 10.1109/ICCV48922.2021.00986http://dx.doi.org/10.1109/ICCV48922.2021.00986]
Ma J Y, Liang P W, Yu W, Chen C, Guo X J, Wu J and Jiang J J. 2020. Infrared and visible image fusion via detail preserving adversarial learning. Information Fusion, 54: 85-98 [DOI: 10.1016/j.inffus.2019.07.005]
Ma J Y, Ma Y and Li C. 2019a. Infrared and visible image fusion methods and applications: a survey. Information Fusion, 45: 153-178 [DOI: 10.1016/j.inffus.2018.02.004]
Ma J Y, Tang L F, Fan F, Huang J, Mei X G and Ma Y. 2022. SwinFusion: cross-domain long-range learning for general image fusion via Swin transformer. IEEE/CAA Journal of Automatica Sinica, 9(7): 1200-1217 [DOI: 10.1109/JAS.2022.105686]
Ma J Y, Yu W, Liang P W, Li C and Jiang J J. 2019b. FusionGAN: a generative adversarial network for infrared and visible image fusion. Information Fusion, 48: 11-26 [DOI: 10.1016/j.inffus.2018.09.004]
Ma J Y, Zhang H, Shao Z F, Liang P W and Xu H. 2021. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion. IEEE Transactions on Instrumentation and Measurement, 70: #5005014 [DOI: 10.1109/TIM.2020.3038013]
Pajares G and de la Cruz J M. 2004. A wavelet-based image fusion tutorial. Pattern Recognition, 37(9): 1855-1872 [DOI: 10.1016/j.patcog.2004.03.010]
Rao Y J. 1997. In-fibre Bragg grating sensors. Measurement Science and Technology, 8(4): 355-375 [DOI: 10.1088/0957-0233/8/4/002]
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2022-05-20].https://arxiv.org/pdf/1409.1556.pdfhttps://arxiv.org/pdf/1409.1556.pdf
Vaswani A,Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł and Polosukhin I. 2017. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, USA: Curran Associates Inc. : 6000-6010
Xu H, Ma J Y, Jiang J J, Guo X J and Ling H B. 2022. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 502-518 [DOI: 10.1109/TPAMI.2020.3012548]
Xu H, Wang X Y and Ma J Y. 2021. DRF: disentangled representation for visible and infrared image fusion. IEEE Transactions on Instrumentation and Measurement, 70: #5006713 [DOI: 10.1109/TIM.2021.3056645]
Xydeas C S and Petrović V. 2000. Objective image fusion performance measure. Electronics Letters, 36(4): 308-309 [DOI: 10.1049/el:20000267]
Zamir S W, Arora A, Khan S, Hayat M, Khan F S and Yang M H. 2022. Restormer: efficient transformer for high-resolution image restoration[EB/OL]. [2022-05-20].https://arxiv.org/pdf/2111.09881.pdfhttps://arxiv.org/pdf/2111.09881.pdf
Zhang Y, Liu Y, Sun P, Yan H, Zhao X L and Zhang L. 2020. IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion, 54: 99-118 [DOI: 10.1016/j.inffus.2019.07.011]
相关文章
相关作者
相关机构
京公网安备11010802024621