深度学习时代图像融合技术进展

左一帆; 方玉明; 马柯德

doi:10.11834/jig.220556

综述 | 浏览量 : 0 下载量: 1422 CSCD: 0

PDF
导出
分享
收藏
专辑

深度学习时代图像融合技术进展
The critical review of the growth of deep learning-based image fusion techniques
2023年28卷第1期页码：102-117
收稿：2022-05-31，

修回：2022-10-9，

录用：2022-10-16，

纸质出版：2023-01-16
DOI： 10.11834/jig.220556
稿件说明：

移动端阅览

左一帆, 方玉明, 马柯德. 深度学习时代图像融合技术进展[J]. 中国图象图形学报, 2023,28(1):102-117. DOI： 10.11834/jig.220556.

Yifan Zuo, Yuming Fang, Kede Ma. The critical review of the growth of deep learning-based image fusion techniques[J]. Journal of Image and Graphics, 2023, 28(1): 102-117. DOI： 10.11834/jig.220556.

摘要

为提升真实场景视觉信号的采集质量，往往需要通过多种融合方式获取相应的图像，例如，多聚焦、多曝光、多光谱和多模态等。针对视觉信号采集的以上特性，图像融合技术旨在利用同一场景不同视觉信号的优势，生成单图像信息描述，提升视觉低、中、高级任务的性能。目前，依托端对端学习强大的特征提取、表征及重构能力，深度学习已成为图像融合研究的主流技术。与传统图像融合技术相比，基于深度学习的图像融合模型性能显著提高。随着深度学习研究的深入，一些新颖的理论和方法也促进了图像融合技术的发展，如生成对抗网络、注意力机制、视觉Transformer和感知损失函数等。为厘清基于深度学习技术的图像融合研究进展，本文首先介绍了图像融合问题建模，并从传统方法视角逐渐向深度学习视角过渡。具体地，从数据集生成、神经网络构造、损失函数设计、模型优化和性能评估等方面总结了基于深度学习的图像融合研究现状。此外，还讨论了选择性图像融合这类衍生问题建模(如基于高分辨率纹理图融合的深度图增强)，回顾了一些基于图像融合实现其他视觉任务的代表性工作。最后，根据现有技术的缺陷，提出目前图像融合技术存在的挑战，并对未来发展趋势给出了展望。

Abstract

To capture more effective visual information of the natural scenes

multi-sensor imaging systems have been challenging in multiple configurations or modalities due to the hardware design constraints. It is required to fuse multiple source images into a single high-quality image in terms of rich and feasible perceptual information and few artifacts. To facilitate various image processing and computer vision tasks

image fusion technique can be used to generate a single and clarified image features. Traditional image fusion models are often constructed in accordance with label-manual features or unidentified feature-learned representations. The generalization ability of the models needs to be developed further.Deep learning technique is focused on progressive multi-layer features extraction via end-to-end model training. Most of demonstration-relevant can be learned for specific task automatically. Compared with the traditional methods

deep learning-based models can improve the fusion performance intensively in terms of image fusion. Current image fusion-related deep learning models are often beneficial based on convolutional neural networks (CNNs) and generative adversarial networks (GANs). In recent years

the newly network structures and training techniques have been incorporated for the growth of image fusion like vision transformers and meta-learning techniques. Most of image fusion-relevant literatures are analyzed from specific multi-fusion issues like exposure

focus

spectrum image

and modality issues. However

more deep learning-related model designs and training techniques is required to be incorporated between multi-fusion tasks. To draw a clear picture of deep learning-based image fusion techniques

we try to review the latest image fusion researches in terms of 1) dataset generation

2) neural network construction

3) loss function design

4) model optimization

and 5) performance evaluation. For dataset generation

we emphasize two categories: a) supervised learning and b) unsupervised (or self-supervised) learning. For neural network construction

we distinguish the early or late stages of this construction process

and the issue of information fusion is implemented between multi-scale

coarse-to-fine and the adversarial networks-incorporated (i.e.

discriminative networks) as well. For loss function design

the perceptual loss functions-specific method is essential for image fusion-related perceptual applications like multi-exposure and multi-focus image fusion. For model optimization

the generic first-order optimization techniques are covered (e.g.

stochastic gradient descent(SGD)

SGD+momentum

Adam

and AdamW) and the advanced alternation and bi-level optimization methods are both taken into consideration. For performance evaluation

a commonly-used quantitative metrics are reviewed for the manifested measurement of fusion performance. The relationship between the loss functions (also as a form of evaluation metrics) are used to drive the learning of CNN-based image fusion methods and the evaluation metrics. In addition

to illustrate the transfer feasibility of image fusion-consensus to a tailored image fusion application

a selection of image fusion methods is discussed (e.g.

a high-quality texture image-fused depth map enhancement). Some popular computer vision tasks are involved in (such as image denoising

blind image deblurring

and image super-resolution)

which can be resolved by image fusion innovatively. Finally

we review some potential challenging issues

including: 1) reliable and efficient ground-truth training data-constructed (i.e.

the input image sequence and the predictable image-fused)

2) lightweight

interpretable

and generalizable CNN-based image fusion methods

3) human or machine-related vision-perceptual calibrated loss functions

4) convergence-accelerated image fusion models in related to adversarial training setting-specific and the bias-related of the test-time training

and 5) human-related ethical issues in relevant to fairness and unbiased performance evaluation.

关键词

Keywords

references

Alparone L, Aiazzi B, Baronti S, Garzelli A, Nencini F and Selva M. 2008. Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering and Remote Sensing, 74(2): 193-200 [DOI: 10.14358/PERS.74.2.193]

Alparone L, Wald L, Chanussot J, Thomas C, Gamba P and Bruce L M. 2007. Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10): 3012-3021 [DOI: 10.1109/TGRS.2007.904923]

Ballester C, Caselles V, Igual L, Verdera J and Rougé B. 2006. A variational model for P+XS image fusion. International Journal of Computer Vision, 69(1): 43-58 [DOI: 10.1007/s11263-006-6852-x]

Barron J T, Adams A, Shih Y and Hernández C. 2015. Fast bilateral-space stereo for synthetic defocus//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 4466-4474 [ DOI: 10.1109/CVPR.2015.7299076 http://dx.doi.org/10.1109/CVPR.2015.7299076 ]

Bhat G, Danelljan M, Van Gool L and Timofte R. 2021. Deep burst super-resolution//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE: 9205-9214 [ DOI: 10.1109/CVPR46437.2021.00909 http://dx.doi.org/10.1109/CVPR46437.2021.00909 ]

Bhatnagar G, Wu Q M J and Liu Z. 2013. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Transactions on Multimedia, 15(5): 1014-1024 [DOI: 10.1109/TMM.2013.2244870]

Cai J R, Gu S H and Zhang L. 2018. Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing, 27(4): 2049-2062 [DOI: 10.1109/TIP.2018.2794218]

Cao P B, Wang Z Y and Ma K D. 2021. Debiased subjective assessment of real-world image enhancement//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE: 711-721 [ DOI: 10.1109/CVPR46437.2021.00077 http://dx.doi.org/10.1109/CVPR46437.2021.00077 ]

Cui G M, Feng H J, Xu Z H, Li Q and Chen Y T. 2015. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition. Optics Communications, 341: 199-209 [DOI: 10.1016/j.optcom.2014.12.032]

Deng X and Dragotti P L. 2021. Deep convolutional neural network for multi-modal image restoration and fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10): 3333-3348 [DOI: 10.1109/TPAMI.2020.2984244]

Ding K Y, Ma K D, Wang S Q and Simoncelli E P. 2021. Comparison of full-reference image quality models for optimization of image processing systems. International Journal of Computer Vision, 129(4): 1258-1281 [DOI: 10.1007/s11263-020-01419-7]

Ding X H, Zhang X Y, Zhou Y Z, Han J G, Ding G G and Sun J. 2022. Scaling up your kernels to 31×31: revisiting large kernel design in CNNs[EB/OL ] . [2022-06-28 ] . http://arxiv.org/pdf/2203.06717.pdf http://arxiv.org/pdf/2203.06717.pdf

Di Zenzo S. 1986. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing, 33(1): 116-125 [DOI: 10.1016/0734-189X(86)90223-9]

Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X H, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J and Houlsby N. 2021. An image is worth 16×16 words: transformers for image recognition at scale [EB/OL ] . [2022-06-28 ] . http://arxiv.org/pdf/2010.11929.pdf http://arxiv.org/pdf/2010.11929.pdf

Dudhane A, Zamir S W, Khan S, Khan F S and Yang M H. 2022. Burst image restoration and enhancement [EB/OL ] . [2022-04-14 ] https://arxiv.org/pdf/2110.03680.pdf https://arxiv.org/pdf/2110.03680.pdf

Fu Y, Zhang T, Zheng Y Q, Zhang D B and Huang H. 2019. Hyperspectral image super-resolution with optimized RGB guidance//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach, USA: IEEE: 11653-11662 [ DOI: 10.1109/CVPR.2019.01193 http://dx.doi.org/10.1109/CVPR.2019.01193 ]

Ghassemian H. 2016. A review of remote sensing image fusion methods. Information Fusion, 32: 75-89 [DOI: 10.1016/j.inffus.2016.03.003]

Guo X P, Nie R C, Cao J D, Zhou D M, Mei L Y and He K J. 2019. FuseGAN: learning to fuse multi-focus image via conditional generative adversarial network. IEEE Transactions on Multimedia, 21(8): 1982-1996 [DOI: 10.1109/TMM.2019.2895292]

Gupta M, Iso D and Nayar S K. 2013. Fibonacci exposure bracketing for high dynamic range imaging//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 1473-1480 [ DOI: 10.1109/ICCV.2013.186 http://dx.doi.org/10.1109/ICCV.2013.186 ]

Han Y, Cai Y Z, Cao Y and Xu X M. 2013. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14(2): 127-135 [DOI: 10.1016/j.inffus.2011.08.002]

He L Z, Zhu H G, Li F, Bai H H, Cong R M, Zhang C J, Lin C Y, Liu M Q and Zhao Y. 2021. Towards fast and accurate real-world depth super-resolution: benchmark dataset and baseline//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE: 9225-9234 [ DOI: 10.1109/CVPR46437.2021.00911 http://dx.doi.org/10.1109/CVPR46437.2021.00911 ]

Hermessi H, Mourali O and Zagrouba E. 2021. Multimodal medical image fusion review: theoretical background and recent advances. Signal Processing, 183: #108036 [DOI: 10.1016/j.sigpro.2021.108036]

Hu J W, Hu P, Kang X D, Zhang H and Fan S S. 2021. Pan-sharpening via multiscale dynamic convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 59(3): 2231-2244 [DOI: 10.1109/TGRS.2020.3007884]

Hui T W, Loy C C and Tang X O. 2016. Depth map super-resolution by deep multi-scale guidance//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 353-369 [ DOI: 10.1007/978-3-319-46487-9_22 http://dx.doi.org/10.1007/978-3-319-46487-9_22 ]

Jin S P, Yu B B, Jing M H, Zhou Y, Liang J J and Ji R H. 2022. DarkVisionNet: low-light imaging via RGB-NIR fusion with deep inconsistency prior. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1): 1104-1112 [DOI: 10.1609/aaai.v36i1.19995]

Jin X, Wang P and Dai Q H. 2020. Parallax tolerant light field stitching for hand-held plenoptic cameras. IEEE Transactions on Image Processing, 29: 1929-1943 [DOI: 10.1109/TIP.2019.2945687]

Jung H, Kim Y, Jang H, Ha N and Sohn K. 2020. Unsupervised deep image fusion with structure tensor representations. IEEE Transactions on Image Processing, 29: 3845-3858 [DOI: 10.1109/TIP.2020.2966075]

Kim B, Ponce J and Ham B. 2021. Deformable kernel networks for joint image filtering. International Journal of Computer Vision, 129(2): 579-600 [DOI: 10.1007/s11263-020-01386-z]

Lan X Y, Ye M, Shao R, Zhong B N, Yuen P C and Zhou H Y. 2019. Learning modality-consistency feature templates: a robust RGB-infrared tracking system. IEEE Transactions on Industrial Electronics, 66(12): 9887-9897 [DOI: 10.1109/TIE.2019.2898618]

Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z and Shi W. 2017. Photo-realistic single image super-resolution using a generative adversarial network//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 105-114 [ DOI: 10.1109/CVPR.2017.19 http://dx.doi.org/10.1109/CVPR.2017.19 ]

Li H and Wu X J. 2019. DenseFuse: a fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5): 2614-2623 [DOI: 10.1109/TIP.2018.2887342]

Li H, Wu X J and Kittler J. 2020a. MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Transactions on Image Processing, 29: 4733-4746 [DOI: 10.1109/TIP.2020.2975984]

Li H F, Cen Y L, Liu Y, Chen X and Yu Z T. 2021. Different input resolutions and arbitrary output resolution: a meta learning-based deep framework for infrared and visible image fusion. IEEE Transactions on Image Processing, 30: 4070-4083 [DOI: 10.1109/TIP.2021.3069339]

Li J X, Guo X B, Lu G M, Zhang B, Xu Y, Wu F and Zhang D. 2020b. DRPL: deep regression pair learning for multi-focus image fusion. IEEE Transactions on Image Processing, 29: 4816-4831 [DOI: 10.1109/TIP.2020.2976190]

Li S T, Kang X D and Hu J W. 2013. Image fusion with guided filtering. IEEE Transactions on Image Processing, 22(7): 2864-2875 [DOI: 10.1109/TIP.2013.2244222]

Li Y J, Huang J B, Ahuja N and Yang M H. 2016. Deep joint image filtering//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 154-169 [ DOI: 10.1007/978-3-319-46493-0_10 http://dx.doi.org/10.1007/978-3-319-46493-0_10 ]

Liu J Y, Fan X, Huang Z B, Wu G Y, Liu R S, Zhong W and Luo Z X. 2022. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE: 5792-5801 [ DOI: 10.1109/CVPR52688.2022.00571 http://dx.doi.org/10.1109/CVPR52688.2022.00571 ]

Liu M W, Wang R H, Li J and Jiao Y Z. 2021. Infrared and visible image fusion with multi-scale anisotropic guided filtering. Journal of Image and Graphics, 26(10): 2421-2432

刘明葳, 王任华, 李静, 焦映臻. 2021. 各向异性导向滤波的红外与可见光图像融合. 中国图象图形学报, 26(10): 2421-2432 [DOI: 10.11834/jig.200339]

Liu Q J, Zhou H Y, Xu Q Z, Liu X Y and Wang Y H. 2021a. PSGAN: a generative adversarial network for remote sensing image pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(12): 10227-10242 [DOI: 10.1109/TGRS.2020.3042974]

Liu Y, Chen X, Peng H and Wang Z F. 2017. Multi-focus image fusion with a deep convolutional neural network. Information Fusion, 36: 191-207 [DOI: 10.1016/j.inffus.2016.12.001]

Liu Z, Lin Y T, Cao Y, Hu H, Wei Y X, Zhang Z, Lin S and Guo B N. 2021b. Swin transformer: hierarchical vision transformer using shifted windows//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE: 9992-10002 [ DOI: 10.1109/ICCV48922.2021.00986 http://dx.doi.org/10.1109/ICCV48922.2021.00986 ]

Loshchilov I and Hutter F. 2017. SGDR: stochastic gradient descent with warm restarts [EB/OL ] . [2022-05-20 ] . http://arxiv.org/pdf/1608.03983.pdf http://arxiv.org/pdf/1608.03983.pdf

Lu K and Zhang L H. 2021. TBEFN: a two-branch exposure-fusion network for low-light image enhancement. IEEE Transactions on Multimedia, 23: 4093-4105 [DOI: 10.1109/TMM.2020.3037526]

Ma J Y, Tang L F, Fan F, Huang J, Mei X G and Ma Y. 2022. SwinFusion: cross-domain long-range learning for general image fusion via Swin transformer. IEEE/CAA Journal of Automatica Sinica, 9(7): 1200-1217 [DOI: 10.1109/JAS.2022.105686]

Ma J Y, Xu H, Jiang J J, Mei X G and Zhang X P. 2020a. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing, 29: 4980-4995 [DOI: 10.1109/TIP.2020.2977573]

Ma J Y, Yu W, Chen C, Liang P W, Guo X J and Jiang J J. 2020b. Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion, 62: 110-120 [DOI: 10.1016/j.inffus.2020.04.006]

Ma J Y, Yu W, Liang P W, Li C and Jiang J J. 2019. FusionGAN: a generative adversarial network for infrared and visible image fusion. Information Fusion, 48: 11-26 [DOI: 10.1016/j.inffus.2018.09.004]

Ma K D, Duanmu Z F, Yeganeh H and Wang Z. 2018. Multi-exposure image fusion by optimizing a structural similarity index. IEEE Transactions on Computational Imaging, 4(1): 60-72 [DOI: 10.1109/TCI.2017.2786138]

Ma K D, Duanmu Z F, Zhu H W, Fang Y M and Wang Z. 2020c. Deep guided learning for fast multi-exposure image fusion. IEEE Transactions on Image Processing, 29: 2808-2819 [DOI: 10.1109/TIP.2019.2952716]

Meng X C, Shen H F, Yuan Q Q, Li H F, Zhang L P and Sun W W. 2019. Pansharpening for cloud-contaminated very high-resolution remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 57(5): 2840-2854 [DOI: 10.1109/TGRS.2018.2878007]

Mildenhall B, Barron J T, Chen J W, Sharlet D, Ng R and Carroll R. 2018. Burst denoising with kernel prediction networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2502-2510 [ DOI: 10.1109/CVPR.2018.00265 http://dx.doi.org/10.1109/CVPR.2018.00265 ]

Minaee S, Boykov Y, Porikli F, Plaza A, Kehtarnavaz N and Terzopoulos D. 2022. Image segmentation using deep learning: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7): 3523-3542 [DOI: 10.1109/TPAMI.2021.3059968]

Mohan S, Vincent J L, Manzorro R, Crozier P, Fernandez-Granda C and Simoncelli E. 2021. Adaptive denoising via gaintuning//Proceedings of the 34th International Conference on Neural Information Processing Systems. [s. l.]: Curran Associates, Inc. : 23727-23740

Park N and Kim S. 2022. How do vision transformers work?[EB/OL ] . [2022-06-28 ] . http://arxiv.org/pdf/2202.06709.pdf http://arxiv.org/pdf/2202.06709.pdf

Petschnigg G, Szeliski R, Agrawala M, Cohen M, Hoppe H and Toyama K. 2004. Digital photography with flash and no-flash image pairs. ACM Transactions on Graphics, 23(3): 664-672 [DOI: 10.1145/1015706.1015777]

Prabhakar K R, Srikar V S and Babu R V. 2017. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs//Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 4724-4732 [ DOI: 10.1109/ICCV.2017.505 http://dx.doi.org/10.1109/ICCV.2017.505 ]

Qu L H, Liu S L, Wang M N and Song Z J. 2022. TransMEF: a transformer-based multi-exposure image fusion framework using self-supervised multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2): 2126-2134 [DOI: 10.1609/aaai.v36i2.20109]

Qu Y, Qi H R and Kwan C. 2018. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2511-2520 [ DOI: 10.1109/CVPR.2018.00266 http://dx.doi.org/10.1109/CVPR.2018.00266 ]

Rahman H, Soundararajan R and Babu R V. 2017. Evaluating multiexposure fusion using image information. IEEE Signal Processing Letters, 24(11): 1671-1675 [DOI: 10.1109/LSP.2017.2752233]

Riegler G, Ferstl D, Rüther M and Bischof H. 2016. A deep primal-dual network for guided depth super-resolution [EB/OL ] . [2022-05-26 ] . http://arxiv.org/pdf/1607.08569.pdf http://arxiv.org/pdf/1607.08569.pdf

Schechner Y Y, Narasimhan S G and Nayar S K. 2001. Instant dehazing of ima ges using polarization//Proceedings of 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Kauai, USA: IEEE: 325-332 [ DOI: 10.1109/CVPR.2001.990493 http://dx.doi.org/10.1109/CVPR.2001.990493 ]

Shang W, Ren D W, Zou D Q, Ren J S, Luo P and Zuo W M. 2021. Bringing events into video deblurring with non-consecutively blurry frames//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE: 4511-4520 [ DOI: 10.1109/ICCV48922.2021.00449 http://dx.doi.org/10.1109/ICCV48922.2021.00449 ]

Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition [EB/OL ] . [2022-09-21 ] . http://arxiv.org/pdf/1409.1556.pdf http://arxiv.org/pdf/1409.1556.pdf

Son C H and Zhang X P. 2016. Layer-based approach for image pair fusion. IEEE Transactions on Image Processing, 25(6): 2866-2881 [DOI: 10.1109/TIP.2016.2556618]

Su H, Jampani V, Sun D Q, Gallo O, Learned-Miller E and Kautz J. 2019. Pixel-adaptive convolutional neural networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 11158-11167 [ DOI: 10.1109/CVPR.2019.01142 http://dx.doi.org/10.1109/CVPR.2019.01142 ]

Su X Y, Li J J and Hua Z. 2022. Transformer-based regression network for pansharpening remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 60: #5407423 [DOI: 10.1109/TGRS.2022.3152425]

Sun Y, Wang X L, Liu Z, Miller J, Efros A A and Hardt M. 2020 Test-time training with self-supervision for generalization under distribution shifts//Proceedings of the 37th International Conference on Machine Learning. [s. l.]: JMLR. org: 9229-9248

Tang L F, Yuan J T and Ma J Y. 2022a. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network. Information Fusion, 82: 28-42 [DOI: 10.1016/j.inffus.2021.12.004]

Tang L F, Yuan J T, Zhang H, Jiang X Y and Ma J Y. 2022b. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 83-84: 79-92 [DOI: 10.1016/j.inffus.2022.03.007]

Vivone G. 2019. Robust band-dependent spatial-detail approaches for panchromatic sharpening. IEEE transactions on Geoscience and Remote Sensing, 57(9): 6421-6433 [DOI: 10.1109/TGRS.2019.2906073]

Wang B, Niu H F, Zeng J C, Bai G F, Lin S Z and Wang Y B. 2021. Latent representation learning model for multi-band images fusion via low-rank and sparse embedding. IEEE Transactions on Multimedia, 23: 3137-3152 [DOI: 10.1109/TMM.2020.3020695]

Wang L, Li B and Tian L F. 2014. EGGDD: an explicit dependency model for multi-modal medical image fusion in shift-invariant shearlet transform domain. Information fusion, 19: 29-37 [DOI: 10.1016/j.inffus.2013.04.005]

Wang L F, Dou J L, Qin P L, Lin S Z, Gao Y and Zhang C C. 2019. Medical image fusion using double dictionary learning and adaptive PCNN. Journal of Image and Graphics, 24(9): 1588-1603

王丽芳, 窦杰亮, 秦品乐, 蔺素珍, 高媛, 张程程. 2019. 双重字典学习与自适应PCNN相结合的医学图像融合. 中国图象图形学报, 24(9): 1588-1603 [DOI: 10.11834/jig.180667]

Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612 [DOI: 10.1109/TIP.2003.819861]

Wang Z X, Zhang J W, Lin M D, Wang J, Luo P and Ren J. 2020. Learning a reinforced agent for flexible exposure bracketing selection//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 1817-1825 [ DOI: 10.1109/CVPR42600.2020.00189 http://dx.doi.org/10.1109/CVPR42600.2020.00189 ]

Wei Q, Bioucas-Dias J, Dobigeon N and Tourneret J Y. 2015. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Transactions on Geoscience and Remote Sensing, 53(7): 3658-3668 [DOI: 10.1109/TGRS.2014.2381272]

Wen Y, Sheng B, Li P, Lin W Y and Feng D D. 2019. Deep color guided coarse-to-fine convolutional network cascade for depth image super-resolution. IEEE Transactions on Image Processing, 28(2): 994-1006 [DOI: 10.1109/TIP.2018.2874285]

Xiao Y, Yang M M, Li C L, Liu L and Tang J. 2022. Attribute-based progressive fusion network for RGBT tracking. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3): 2831-2838 [DOI: 10.1609/aaai.v36i3.20187]

Xie Q, Zhou M H, Zhao Q, Xu Z B and Meng D Y. 2022. MHF-net: an interpretable deep network for multispectral and hyperspectral image fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3): 1457-1473 [DOI: 10.1109/TPAMI.2020.3015691]

Xie W Y, Cui Y H, Li Y S, Lei J, Du Q and Li J J. 2021. HPGAN: hyperspectral pansharpening using 3-D generative adversarial networks. IEEE Transactions on Geoscience and Remote Sensing, 59(1): 463-477 [DOI: 10.1109/TGRS.2020.2994238]

Xu H and Ma J Y. 2021. EMFusion: an unsupervised enhanced medical image fusion network. Information Fusion, 76: 177-186 [DOI: 10.1016/j.inffus.2021.06.001]

Xu H, Ma J Y, Jiang J J, Guo X J and Ling H B. 2022a. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 502-518 [DOI: 10.1109/TPAMI.2020.3012548]

Xu H, Ma J Y, Shao Z F, Zhang H, Jiang J J and Guo X J. 2021. SDPNet: a deep network for pan-sharpening with enhanced information representation. IEEE Transactions on Geoscience and Remote Sensing, 59(5): 4120-4134 [DOI: 10.1109/TGRS.2020.3022482]

Xu H, Ma J Y, Yuan J T, Le Z L and Liu W. 2022b. RFNet: unsupervised network for mutually reinforcing multi-modal image registration and fusion//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE: 19647-19656 [ DOI: 10.1109/CVPR52688.2022.01906 http://dx.doi.org/10.1109/CVPR52688.2022.01906 ]

Xu S, Amira O, Liu J M, Zhang C X, Zhang J S and Li G H. 2020. HAM-MFN: hyperspectral and multispectral image multiscale fusion network with RAP loss. IEEE Transactions on Geoscience and Remote Sensing, 58(7): 4618-4628 [DOI: 10.1109/TGRS.2020.2964777]

Yang J F, Fu X Y, Hu Y W, Huang Y, Ding X H and Paisley J. 2017. PanNet: a deep network architecture for pan-sharpening//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 1753-1761[ DOI: 10.1109/ICCV.2017.193 http://dx.doi.org/10.1109/ICCV.2017.193 ]

Yang S Y, Wang M, Jiao L C, Wu R X and Wang Z X. 2010. Image fusion based on a new contourlet packet. Information Fusion, 11(2): 78-84 [DOI: 10.1016/j.inffus.2009.05.001]

Yao J, Hong D F, Chanussot J, Meng D Y, Zhu X X and Xu Z B. 2020. Cross-attention in coupled unmixing nets for unsupervised hyperspectral super-resolution//Proceedings of the 16th European Conference on Computer Vision. Glasgow, UK: Springer: 208-224 [ DOI: 10.1007/978-3-030-58526-6_13 http://dx.doi.org/10.1007/978-3-030-58526-6_13 ]

Ye X C, Sun B L, Wang Z H, Yang J Y, Xu R, Li H J and Li B P. 2020. PMBANet: progressive multi-branch aggregation network for scene depth super-resolution. IEEE Transactions on Image Processing, 29: 7427-7442 [DOI: 10.1109/TIP.2020.3002664]

Yokoya N, Yairi T and Iwasaki A. 2012. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2): 528-537 [DOI: 10.1109/TGRS.2011.2161320]

Zhang H and Ma J Y. 2021a. SDNet: a versatile squeeze-and-decomposition network for real-time image fusion. International Journal of Computer Vision, 129(10): 2761-2785[DOI: 10.1007/s11263-021-01501-8]

Zhang H, Xu H, Xiao Y, Guo X J and Ma J Y. 2020a. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. Proceedings of the AAAI Conference on Artificial Intelligence, 34(7): 12797-12804 [DOI: 10.1609/aaai.v34i07.6975]

Zhang K H, Liu Q S, Wu Y and Yang M H. 2016. Robust visual tracking via convolutional networks without training. IEEE Transactions on Image Processing, 25(4): 1779-1792 [DOI: 10.1109/TIP.2016.2531283]

Zhang X T, Huang W, Wang Q and Li X L. 2021b. SSR-NET: spatial-spectral reconstruction network for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing, 59(7): 5953-5965 [DOI: 10.1109/TGRS.2020.3018732]

Zhang Y, Liu Y, Sun P, Yan H, Zhao X L and Zhang L. 2020b. IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion, 54: 99-118 [DOI: 10.1016/j.inffus.2019.07.011]

Zhao W D, Wang D and Lu H C. 2019. Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 29(4): 1102-1115 [DOI: 10.1109/TCSVT.2018.2821177]

Zhao Z X, Xu S, Zhang C X, Liu J M, Zhang J S and Li P F. 2021. DIDFuse: deep image decomposition for infrared and visible image fusion//Proceedings of the 29th International Joint Conference on Artificial Intelligence. Yokohama, Japan: [s. n. ] : 970-976 [ DOI: 10.24963/ijcai.2020/135 http://dx.doi.org/10.24963/ijcai.2020/135 ]

Zheng K, Gao L R, Liao W Z, Hong D F, Zhang B, Cui X M and Chanussot J. 2021. Coupled convolutional neural network with adaptive response function learning for unsupervised hyperspectral super resolution. IEEE Transactions on Geoscience and Remote Sensing, 59(3): 2487-2502 [DOI: 10.1109/TGRS.2020.3006534]

Zheng M Y, Qi G Q, Zhu Z Q, Li Y Y, Wei H Y and Liu Y. 2020. Image dehazing by an artificial image fusion method based on adaptive structure decomposition. IEEE Sensors Journal, 20(14): 8062-8072 [DOI: 10.1109/JSEN.2020.2981719]

Zhu M F, Pan P B, Chen W and Yang Y. 2020. EEMEFN: low-light image enhancement via edge-enhanced multi-exposure fusion network.Proceedings of the AAAI Conference on Artificial Intelligence, 34(7): 13106-13113 [DOI: 10.1609/aaai.v34i07.7013]

Zhu X X and Bamler R. 2013. A sparse image fusion algorithm with application to pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 51(5): 2827-2836 [DOI: 10.1109/TGRS.2012.2213604]

Zhu Y B, Li C L, Luo B, Tang J and Wang X. 2019. Dense feature aggregation and pruning for RGBT tracking//Proceedings of the 27th ACM International Conference on Multimedia. Nice, France: ACM: 465-472 [ DOI: 10.1145/3343031.3350928 http://dx.doi.org/10.1145/3343031.3350928 ]

Zuo Y F, Fang Y M, An P, Shang X W and Yang J N. 2021. Frequency-dependent depth map enhancement via iterative depth-guided affine transformation and intensity-guided refinement. IEEE Transactions on Multimedia, 23: 772-783 [DOI: 10.1109/TMM.2020.2987706]

Zuo Y F, Wu Q, Fang Y M, An P, Huang L Q andChen Z F. 2020. Multi-scale frequency reconstruction for guided depth map super-resolution via deep residual network. IEEE Transactions on Circuits and Systems for Video Technology, 30(2): 297-306 [DOI: 10.1109/TCSVT.2018.2890271]