浏览全部资源
扫码关注微信
收稿日期：2021-07-21，
修回日期：2021-09-10，
录用日期：2021-9-17，
纸质出版日期：2022-11-16
移动端阅览
目的
2
红外与可见光图像融合的目标是将红外图像与可见光图像的互补信息进行融合，增强源图像中的细节场景信息。然而现有的深度学习方法通常人为定义源图像中需要保留的特征，降低了热目标在融合图像中的显著性。此外，特征的多样性和难解释性限制了融合规则的发展，现有的融合规则难以对源图像的特征进行充分保留。针对这两个问题，本文提出了一种基于特有信息分离和质量引导的红外与可见光图像融合算法。
方法
2
本文提出了基于特有信息分离和质量引导融合策略的红外与可见光图像融合算法。设计基于神经网络的特有信息分离以将源图像客观地分解为共有信息和特有信息，对分解出的两部分分别使用特定的融合策略；设计权重编码器以学习质量引导的融合策略，将衡量融合图像质量的指标应用于提升融合策略的性能，权重编码器依据提取的特有信息生成对应权重。
结果
2
实验在公开数据集RoadScene上与6种领先的红外与可见光图像融合算法进行了对比。此外，基于质量引导的融合策略也与4种常见的融合策略进行了比较。定性结果表明，本文算法使融合图像具备更显著的热目标、更丰富的场景信息和更多的信息量。在熵、标准差、差异相关和、互信息及相关系数等指标上，相较于对比算法中的最优结果分别提升了0.508%、7.347%、14.849%、9.927%和1.281%。
结论
2
与具有领先水平的红外与可见光算法以及现有的融合策略相比，本文融合算法基于特有信息分离和质量引导，融合结果具有更丰富的场景信息、更强的对比度，视觉效果更符合人眼的视觉特征。
Objective
2
Infrared and visible image fusion is essential to computer vision and image processing. To strengthen the scenes recognition derived of multisource images
more multi-sensors imagery information is required to be fused in relation to infrared and visible images. A fused image is generated for human perception-oriented visual tasks like video surveillance
target recognition and scene understanding. However
the existing fusion methods are usually designed by manually selecting the characteristics to be preserved. The existing fusion methods can be roughly divided into two categories in the context of traditional fusion methods and the deep learning-based fusion methods. For the traditional methods
to comprehensively characterize and decompose the source images
they need to manually design transformation methods. The fusion strategies are manually designed to fuse the decomposed subparts. The manually designed decomposition methods become more and more complex
which leads to the decline of fusion efficiency. For the deep learning-based methods
some methods define the unique characteristics of source images via human observation. The fused images are expected to preserve these characteristics as much as possible. However
it is difficult and unsuitable to identify the vital information through one or a few characteristics. Other methods are focused on preserving higher structural similarity with source images in terms of the fused image. It will reduce the saliency of thermal targets in the fusion result
which is not conductive to the rapid location and capture of thermal targets by the human vision system. Our method is designed to solve these two issues. We develop a new deep learning-based decomposition method for infrared and visible image fusion. Besides
we propose a deep learning-based and quality-guided fusion strategy to fuse the decomposed parts.
Method
2
Our infrared and visible image fusion method is based on the information decomposed and quality-guided fusion strategy. First
we design an image decomposition and representation way
which is based on the convolution neural network (CNN). For each source image
two encoders are used to decompose the source images into the common part and unique part. Based on three loss functions (including a reconstruction loss
a translation loss and a loss to constrain the unique information)
four encoders are learned to realize the physical-meaning-related decomposition. For the two decomposed parts
specific fusion strategies are applied for each. For the common parts
a traditional fusion strategy is selected to reduce the computational complexity and improve the fusion efficiency. For the unique parts
a weight encoder is assigned to learn the quality-guided fusion strategy
which can further preserve the complementary information derived from multi-source images. To improve the fusion performance
the metrics is used to evaluate the quality of fused images. The weight encoder generates the corresponding weights according to the unique information. The generator-optimized in the unique information decomposition procedure is used to generate the final fused image according to the fused common part and unique part.
Result
2
Our method is compared to six state-of-the-art visible and infrared image fusion methods on the publicly available dataset named as RoadScene. In addition
the quality-guided fusion strategy is also in compared with four common fusion strategies
including mean
max
addition and
$$l_1$$
-norm on the publicly available dataset. The qualitative comparisons show that our fusion results have three priorities as mentioned below: first
our fusion results can highlight thermal targets. It is beneficial to capture the thermal targets in accordance with the high contrast. Second
more scene information and clearer edges or textures can be presented. Some regions and textures are enhanced as well. Third
even in some extreme cases
our fusion results also show the most information. The effective information in one source image is preserved in the fused image without being affected by the regions in the other source image which has less information. Additionally
we also perform the quantitative evaluation of the proposed method with comparative fusion methods and strategies on six objective metrics. These metrics is composed of entropy
standard deviation
the sum of difference correlation
mutual information and correlation coefficient. Our method shows the best or comparable performance. Compared to existing fusion methods
our average is increased by 0.508%
7.347%
14.849%
9.927% and 1.281% compared with existing methods
respectively. Furthermore
our method is applied to fuse a RGB visible image and a single-channel infrared image. The results show that our method is feasible to improve fusion results.
Conclusion
2
We develop an infrared and visible image fusion method based on information decomposition and quality-guided fusion strategy. The experiment results show that the proposed fusion method and fusion strategy outperforms several state-of-the-art infrared and visible image fusion methods and the existing fusion strategies. Both the qualitative and quantitative results show the effectiveness of the proposed method and strategy.
Aslantas V and Bendes E. 2015. A new image quality metric for image fusion: the sum of the correlations of differences. AEU-International Journal of Electronics and Communications, 69(12): 1890-1896[DOI: 10.1016/j.aeue.2015.09.004]
Bavirisetti D P, Xiao G and Liu G. 2017. Multi-sensor image fusion based on fourth order partial differential equations//Proceedings of the 20th International Conference on Information Fusion. Xi'an, China: IEEE: 1-9[ DOI: 10.23919/ICIF.2017.8009719 http://dx.doi.org/10.23919/ICIF.2017.8009719 ]
Chen J, Li X J, Luo L B, Mei X G and Ma J Y. 2020. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Information Sciences, 508: 64-78[DOI: 10.1016/j.ins.2019.08.066]
Chen M S. 2016. Image fusion of visual and infrared image based on NSCT and compressed sensing. Journal of Image and Graphics, 21(1): 39-44
陈木生. 2016. 结合NSCT和压缩感知的红外与可见光图像融合. 中国图象图形学报, 21(1): 39-44[DOI: 10.11834/jig.20160105]
Du Q L, Xu H, Ma Y, Huang J and Fan F. 2018. Fusing infrared and visible images of different resolutions via total variation model. Sensors, 18(11): #3827[DOI: 10.3390/s18113827]
Gong R and Wang X C. 2019. Infrared and visible image fusion based on BEMD and W-transform. Journal of Image and Graphics, 24(6): 987-999
宫睿, 王小春. 2019. BEMD分解和W变换相结合的红外与可见光图像融合. 中国图象图形学报, 24(6): 987-999[DOI: 10.11834/jig.180530]
Hou R C, Zhou D M, Nie R C, Liu D, Xiong L, Guo Y B and Yu C B. 2020. VIF-Net: an unsupervised framework for infrared and visible image fusion. IEEE Transactions on Computational Imaging, 6: 640-651[DOI: 10.1109/TCI.2020.2965304]
Li H and Wu X J. 2019. DenseFuse: a fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5): 2614-2623[DOI: 10.1109/TIP.2018.2887342]
Li H, Wu X J and Kittler J. 2020a. MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Transactions on Image Processing, 29: 4733-4746[DOI: 10.1109/TIP.2020.2975984]
Li J, Huo H T, Li C, Wang R H and Feng Q. 2020b. AttentionFGAN: infrared and visible image fusion using attention-based generative adversarial networks. IEEE Transactions on Multimedia, 23: 1383-1396[DOI: 10.1109/TMM.2020.2997127]
Liu Y, Liu S P and Wang Z F. 2015. A general framework for image fusion based on multi-scale transform and sparse representation. Information Fusion, 24: 147-164[DOI: 10.1016/j.inffus.2014.09.004]
Ma J L, Zhou Z Q, Wang B and Zong H. 2017. Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Physics and Technology, 82: 8-17[DOI: 10.1016/j.infrared.2017.02.005]
Ma J Y, Chen C, Li C and Huang J. 2016. Infrared and visible image fusion via gradient transfer and total variation minimization. Information Fusion, 31: 100-109[DOI: 10.1016/j.inffus.2016.02.001]
Ma J Y, Xu H, Jiang J J, Mei X G and Zhang X P. 2020. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing, 29: 4980-4995[DOI: 10.1109/TIP.2020.2977573]
Ma J Y, Yu W, Liang P W, Li C and Jiang J J. 2019. FusionGAN: a generative adversarial network for infrared and visible image fusion. Information Fusion, 48: 11-26[DOI: 10.1016/j.inffus.2018.09.004]
Roberts J W, Van Aardt J A and Ahmed F B. 2008. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2(1): #023522[DOI: 10.1117/1.2945910]
Singh R, Vatsa M and Noore A. 2008. Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition. Pattern Recognition, 41(3): 880-893[DOI: 10.1016/j.patcog.2007.06.022]
Wang J, Peng J Y, Feng X Y, He G Q and Fan J P. 2014. Fusion method for infrared and visible images by using non-negative sparse representation. Infrared Physics and Technology, 67: 477-489[DOI: 10.1016/j.infrared.2014.09.019]
Xu H, Ma J Y, Jiang J J, Guo X J and Ling H B. 2020. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 502-518[DOI: 10.1109/TPAMI.2020.3012548]
Xydeas C S and Petrović V. 2000. Objective image fusion performance measure. Electronics Letters, 36(4): 308-309[DOI: 10.1049/el:20000267]
Yang Y, Tong S and Huang S Y. 2015. Image fusion based on fast discrete Curvelet transform. Journal of Image and Graphics, 20(2): 219-228
杨勇, 童松, 黄淑英. 2015. 快速离散Curvelet变换域的图像融合. 中国图象图形学报, 20(2): 219-228[DOI: 10.11834/jig.20150208]
Yin M, Duan P H, Liu W and Liang X Y. 2017. A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation. Neurocomputing, 226: 182-191[DOI: 10.1016/j.neucom.2016.11.051]
Zhang Q, Huang N C, Yao L, Zhang D W, Shan C F and Han J G. 2019. RGB-T salient object detection via fusing multi-level CNN features. IEEE Transactions on Image Processing, 29: 3321-3335[DOI: 10.1109/TIP.2019.2959253]
Zhang Y, Liu Y, Sun P, Yan H, Zhao X L and Zhang L. 2020. IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion, 54: 99-118[DOI: 10.1016/j.inffus.2019.07.011]
Zhou H B, Hou J L, Wu W, Zhang Y D, Wu Y T and Ma J Y. 2021. Infrared and visible image fusion based on semantic segmentation. Journal of Computer Research and Development, 58(2): 436-443
周华兵, 侯积磊, 吴伟, 张彦铎, 吴云韬, 马佳义. 2021. 基于语义分割的红外和可见光图像融合. 计算机研究与发展, 58(2): 436-443[DOI: 10.7544/issn1000-1239.2021.20200244]
Zhou Z Q, Wang B, Li S and Dong M J. 2016. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters. Information Fusion, 30: 15-26[DOI: 10.1016/j.inffus.2015.11.003]
Zhu Z Q, Yin H P, Chai Y, Li Y X and Qi G Q. 2018. A novel multi-modality image fusion method based on image decomposition and sparse representation. Information Sciences, 432: 516-529[DOI: 10.1016/j.ins.2017.09.010]
相关作者
相关机构
京公网安备11010802024621