深度学习多聚焦图像融合方法综述

王磊; 齐争争; 刘羽

doi:10.11834/jig.220593

综述 | 浏览量 : 0 下载量: 871 CSCD: 0

PDF
导出
分享
收藏
专辑

深度学习多聚焦图像融合方法综述
The review of multi-focus image fusion methods based on deep learning
2023年28卷第1期页码：80-101
收稿：2022-06-07，

修回：2022-7-21，

录用：2022-7-28，

纸质出版：2023-01-16
DOI： 10.11834/jig.220593
稿件说明：

移动端阅览

王磊, 齐争争, 刘羽. 深度学习多聚焦图像融合方法综述[J]. 中国图象图形学报, 2023,28(1):80-101. DOI： 10.11834/jig.220593.

Lei Wang, Zhengzheng Qi, Yu Liu. The review of multi-focus image fusion methods based on deep learning[J]. Journal of Image and Graphics, 2023, 28(1): 80-101. DOI： 10.11834/jig.220593.

摘要

多聚焦图像融合是一种以软件方式有效扩展光学镜头景深的技术，该技术通过综合同一场景下多幅部分聚焦图像包含的互补信息，生成一幅更加适合人类观察或计算机处理的全聚焦融合图像，在数码摄影、显微成像等领域具有广泛的应用价值。传统的多聚焦图像融合方法往往需要人工设计图像的变换模型、活跃程度度量及融合规则，无法全面充分地提取和融合图像特征。深度学习由于强大的特征学习能力被引入多聚焦图像融合问题研究，并迅速发展为该问题的主流研究方向，多种多样的方法不断提出。鉴于国内鲜有多聚焦图像融合方面的研究综述，本文对基于深度学习的多聚焦图像融合方法进行系统综述，将现有方法分为基于深度分类模型和基于深度回归模型两大类，对每一类中的代表性方法进行介绍；然后基于3个多聚焦图像融合数据集和8个常用的客观质量评价指标，对25种代表性融合方法进行了性能评估和对比分析；最后总结了该研究方向存在的一些挑战性问题，并对后续研究进行展望。本文旨在帮助相关研究人员了解多聚焦图像融合领域的研究现状，促进该领域的进一步发展。

Abstract

Multi-focus image fusion technique can extend the depth-of-field (DOF) of optical lenses effectively via a software-based manner. It aims to fuse a set of partially focused source images of the same scene by generating an all-in-focus fused image

which will be more suitable for human or machine perception. As a result

multi-focus image fusion is of high practical significance in many areas including digital photography

microscopy imaging

integral imaging

thermal imaging

etc. Traditional multi-focus image fusion methods

which generally include transform domain methods (e.g.

multi-scale transform-based methods and sparse representation-based methods) and spatial domain methods (e.g.

block-based methods and pixel-based methods)

are often based on manually designed transform models

activity level measures and fusion rules. To achieve high fusion performance

these key factors tend to become much more complicated in the fusion algorithm

which are usually at the cost of computational efficiency. In addition

these key factors are often independently designed with relatively weak association

which limits the fusion performance to a great extent. In the past few years

deep learning has been introduced into the study of multi-focus image fusion and has rapidly emerged as the current mainstream of this field

with a variety of deep learning-based fusion methods being proposed in the literature. Deep learning models like convolutional neural networks (CNNs) and generative adversarial networks (GANs) have been facilitating in the study of multi-focus image fusion. It is of high significance to conduct a comprehensive survey to review the recent advances achieved in deep learning-based multi-focus image fusion and put forward some future prospects for further improvement. Some survey papers related to image fusion including multi-focus image fusion have been recently published in the international journals around 2020. However

the survey works on multi-focus image fusion are rarely reported in Chinese journals. Moreover

considering that this field grows very rapidly with dozens of papers being published each year

a more timely survey is also highly expected. Based on the above considerations

we demonstrate a systematic review for the deep learning-based multi-focus image fusion methods. In this paper

the existing deep learning-based methods are classified into two main categories: 1) deep classification model-based methods and 2) deep regression model-based methods. Additionally

these two categories of methods are further divided into sub-categories. Specifically

the classification model-based methods are further divided into image block-based methods and image segmentation-based methods in terms of the pixel processing manner adopted. The regression model-based methods are further divided into supervised learning-based methods and unsupervised learning-based methods

according to the learning manner of network models. For each category

the representative fusion methods are introduced as well. In addition

we conduct a comparative study on the performance of 25 representative multi-focus image fusion methods

including 5 traditional transform domain methods

5 traditional spatial domain methods and 15 deep learning-based methods. To this end

we use three commonly-used multi-focus image fusion datasets in the experiments including "Lytro"

"MFFW" and "Classic". Additionally

eight objective evaluation metrics that are widely used in multi-focus image fusion are adopted for performance assessment

which are composed of include two information theory-based metrics

two image feature-based metrics

two structural similarity-based metrics and two human visual perception-based metrics. The experimental results verify that deep learning-based methods can achieve very promising fusion results. However

it is worth noting that the performance of most deep learning-based methods is not significantly better than that of the traditional fusion methods. One main reason for this phenomenon is the lack of large-scale and realistic datasets for training in multi-focus image fusion

and the way to create synthetic datasets for training is inevitability different from the real situation

leading to that the potential of deep learning-based methods cannot be fully tapped. Finally

we summarize some challenging problems in the study of deep learning-based multi-focus image fusion and put forward some future prospects accordingly

which mainly include the four aspects as following: 1) the fusion of focus boundary regions; 2) the fusion of mis-registered source images; 3) the construction of large-scale datasets with real labels for network training; and 4) the improvement of network architecture and model training approach.

关键词

Keywords

references

Amin-Naji M, Aghagolzadeh A and Ezoji M. 2019. Ensemble of CNN for multi-focus image fusion. Information Fusion, 51: 201-214 [DOI: 10.1016/j.inffus.2019.02.003]

Amin-Naji M, Aghagolzadeh A and Ezoji M. 2020. CNNs hard voting for multi-focus image fusion. Journal of Ambient Intelligence and Humanized Computing, 11(4): 1749-1769 [DOI: 10.1007/s12652-019-01199-0]

Aslantas V and Kurban R. 2010. Fusion of multi-focus images using differential evolution algorithm. Expert Systems with Applications, 37(12): 8861-8870 [DOI: 10.1016/j.eswa.2010.06.011]

Bai X Z, Zhang Y, Zhou F G and Xue B D. 2015. Quadtree-based multi-focus image fusion using a weighted focus-measure. Information Fusion, 22: 105-118 [DOI: 10.1016/j.inffus.2014.05.003]

Burt P J and Adelson E H. 1985. Merging images through pattern decomposition//Proceedings Volume 0575, Applications of Digital Image Processing Ⅷ. San Diego, USA: SPIE: #966501 [ DOI: 10.1117/12.966501 http://dx.doi.org/10.1117/12.966501 ]

Chen Y and Blum R S. 2009. A new automated quality assessment algorithm for image fusion. Image and Vision Computing, 27(10): 1421-1432 [DOI: 10.1016/j.imavis.2007.12.002]

Cheng C Y, Wu X J, Xu T Y and Chen G Y. 2021. UNIFusion: a lightweight unified image fusion network. IEEE Transactions on Instrumentation and Measurement, 70: #5016614 [DOI: 10.1109/TIM.2021.3109379]

Deng X and Dragotti P L. 2021. Deep convolutional neural network for multi-modal image restoration and fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10): 3333-3348 [DOI: 10.1109/TPAMI.2020.2984244]

Deshmukh V, Khaparde A and Shaikh S. 2018. Multi-focus image fusion using deep belief network//Proceedings of Information and Communication Technology for Intelligent Systems (ICTIS 2017). Ahmedabad, India: Springer: 233-241 [ DOI: 10.1007/978-3-319-63673-3_28 http://dx.doi.org/10.1007/978-3-319-63673-3_28 ]

Duan J W, Chen L and Chen C L P. 2018. Multifocus image fusion with enhanced linear spectral clustering and fast depth map estimation. Neurocomputing, 318: 43-54 [DOI: 10.1016/j.neucom.2018.08.024]

Duan Z, Zhang T P, Luo X L and Tan J. 2021. DCKN: multi-focus image fusion via dynamic convolutional kernel network. Signal Processing, 189: #108282 [DOI: 10.1016/j.sigpro.2021.108282]

Gao W C, Yu L, Tan Y and Yang P N. 2022. MSIMCNN: multi-scale inception module convolutional neural network for multi-focus image fusion. Applied Intelligence, 52: 14085-14100 [DOI: 10.1007/s10489-022-03160-9]

Guan Z, Wang X, Nie R C, Yu S S and Wang C C. 2022. NCDCN: multi-focus image fusion via nest connection and dilated convolution network. Applied Intelligence: #03194z

Guo D, Yan J W and Qu X B. 2015. High quality multi-focus image fusion using self-similarity and depth information. Optics Communications, 338: 138-144 [DOI: 10.1016/j.optcom.2014.10.031]

Guo X P, Mei L Y and Nie R C. 2018a. Learning to fuse multi-focus image via convolutional network modeling//Proceedings of the 7th International Conference on Informatics, Environment, Energy and Applications. Beijing, China: ACM: 236-241 [ DOI: 10.1145/3208854.3208896 http://dx.doi.org/10.1145/3208854.3208896 ]

Guo X P, Meng L Y, Mei L Y, Weng Y Y and Tong H Q. 2020. Multi-focus image fusion with Siamese self-attention network. IET Image Processing, 14(7): 1339-1346 [DOI: 10.1049/iet-ipr.2019.0883]

Guo X P, Nie R C, Cao J D, Zhou D M, Mei L Y and He K J. 2019. FuseGAN: learning to fuse multi-focus image via conditional generative adversarial network. IEEE Transactions on Multimedia, 21(8): 1982-1996 [DOI: 10.1109/TMM.2019.2895292]

Guo X P, Nie R C, Cao J D, Zhou D M and Qian W H. 2018b. Fully convolutional network-based multifocus image fusion. Neural Computation, 30(7): 1775-1800 [DOI: 10.1162/neco_a_01098]

Han Y, Cai Y Z, Cao Y and Xu X M. 2013. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14(2): 127-135 [DOI: 10.1016/j.inffus.2011.08.002]

Hong X X and Kintak U. 2019. Multi-focus image fusion algorithm based on non-uniform rectangular partition and generative adversarial network//Proceedings of 2019 International Conference on Wavelet Analysis and Pattern Recognition (ICWAPR). Kobe,Japan: IEEE: 1-6 [ DOI: 10.1109/ICWAPR48189.2019.8946467 http://dx.doi.org/10.1109/ICWAPR48189.2019.8946467 ]

Hossny M, Nahavandi S and Creighton D. 2008. Comments on Information measure for performance of image fusion. Electronics Letters, 44(18): 1066-1067 [DOI: 10.1049/el:20081754]

Huang J, Le Z L, Ma Y, Mei X G and Fan F. 2020. A generative adversarial network with adaptive constraints for multi-focus image fusion. Neural Computing and Applications, 32(18): 15119-15129 [DOI: 10.1007/s00521-020-04863-1]

Jiang L M, Fan H and Li J J. 2022. A multi-focus image fusion method based on attention mechanism and supervised learning. Applied Intelligence, 52(1): 339-357 [DOI: 10.1007/s10489-021-02358-7]

Jiang L M, Fan H, Li J J and Tu C H. 2021. Pseudo-Siamese residual atrous pyramid network for multi-focus image fusion. IET Image Processing, 15(13): 3304-3317 [DOI: 10.1049/ipr2.12326]

Jiang Y and Wang M H. 2014. Image fusion with morphological component analysis. Information Fusion, 18: 107-118 [DOI: 10.1016/j.inffus.2013.06.001]

Jung H, Kim Y, Jang H, Ha N and Sohn K. 2020. Unsupervised deep image fusion with structure tensor representations. IEEE Transactions on Image Processing, 29: 3845-3858 [DOI: 10.1109/TIP.2020.2966075]

Lahoud F and Süsstrunk S. 2019. Fast and efficient zero-learning image fusion [EB/OL ] . [2022-05-09 ] . https://arxiv.org/pdf/1905.03590.pdf https://arxiv.org/pdf/1905.03590.pdf

Lai R, Li Y X, Guan J T and Xiong A. 2019. Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access, 7: 114385-114399 [DOI: 10.1109/ACCESS.2019.2935006]

Lewis J J, O'callaghan R J, Nikolov S G, Bull D R and Canagarajah N. 2007. Pixel- and region-based image fusion with complex wavelets. Information Fusion, 8(2): 119-130 [DOI: 10.1016/j.inffus.2005.09.006]

Li H, Manjunath B S and Mitra S K. 1995. Multisensor image fusion using the wavelet transform. Graphical Models and Image Processing, 57(3): 235-245 [DOI: 10.1006/gmip.1995.1022]

Li H, Zhang L M, Jiang M R and Li Y L. 2021. Multi-focus image fusion algorithm based on supervised learning for fully convolutional neural network. Pattern Recognition Letters, 141: 45-53 [DOI: 10.1016/j.patrec.2020.11.014]

Li H G, Nie R C, Cao J D, Guo X P, Zhou D M and He K J. 2019a. Multi-focus image fusion using U-shaped networks with a hybrid objective. IEEE Sensors Journal, 19(21): 9755-9765 [DOI: 10.1109/JSEN.2019.2928818]

Li H G, Nie R C, Zhou D M and Gou X P. 2018. Convolutional neural network based multi-focus image fusion//Proceedings of the 2nd I nternational Conference on Algorithms, Computing and Systems. Beijing, China: IEEE: 148-154 [ DOI: 10.1145/3242840.3242863 http://dx.doi.org/10.1145/3242840.3242863 ]

Li J J, Yuan G J and Fan H. 2019b. Multifocus image fusion using wavelet-domain-based deep CNN. Computational Intelligence and Neuroscience, 2019: #4179397 [DOI: 10.1155/2019/4179397]

Li J X, Guo X B, Lu G M, Zhang B, Xu Y, Wu F and Zhang D. 2020. DRPL: deep regression pair learning for multi-focus image fusion. IEEE Transactions on Image Processing, 29: 4816-4831 [DOI: 10.1109/TIP.2020.2976190]

Li M, Cai W and Tan Z. 2006. A region-based multi-sensor image fusion scheme using pulse-coupled neural network. Pattern Recognition Letters, 27(16): 1948-1956 [DOI: 10.1016/j.patrec.2006.05.004]

Li S T, Kang X D, Fang L Y, Hu J W and Yin H T. 2017. Pixel-level image fusion: a survey of the state of the art. Information Fusion, 33: 100-112 [DOI: 10.1016/j.inffus.2016.05.004]

Li S T, Kang X D and Hu J W. 2013a. Image fusion with guided filtering. IEEE Transactions on Image Processing, 22(7): 2864-2875 [DOI: 10.1109/TIP.2013.2244222]

Li S T, Kang X D, Hu J W and Yang B. 2013b. Image matting for fusion of multi-focus images in dynamic scenes. Information Fusion, 14(2): 147-162 [DOI: 10.1016/j.inffus.2011.07.001]

Li S T, Kwok J T and Wang Y N. 2001. Combination of images with diverse focuses using the spatial frequency. Information Fusion, 2(3): 169-176 [DOI: 10.1016/S1566-2535(01)00038-0]

Li S T and Yang B. 2008a. Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recognition Letters, 29(9): 1295-1301 [DOI: 10.1016/j.patrec.2008.02.002]

Li S T and Yang B. 2008b. Multifocus image fusion using region segmentation and spatial frequency. Image and Vision Computing, 26(7): 971-979 [DOI: 10.1016/j.imavis.2007.10.012]

Liang J L, He Y, Liu D and Zeng X J. 2012. Image fusion using higher order singular value decomposition. IEEE Transactions on Image Processing, 21(5): 2898-2909 [DOI: 10.1109/TIP.2012.2183140]

Liu S Q, Ma J, Yang Y, Qiu T, Li H L, Hu S H and Zhang Y D. 2022. A multi-focus color image fusion algorithm based on low vision image reconstruction and focused feature extraction. Signal Processing: Image Communication, 100: #116533 [DOI: 10.1016/j.image.2021.116533]

Liu Y, Chen X, Peng H and Wang Z F. 2017. Multi-focus image fusion with a deep convolutional neural network. Information Fusion, 36: 191-207 [DOI: 10.1016/j.inffus.2016.12.001]

Liu Y, Chen X, Wang Z F, Wang Z J, Ward R K and Wang X S. 2018. Deep learning for pixel-level image fusion: recent advances and future prospects. Information Fusion, 42: 158-173 [DOI: 10.1016/j.inffus.2017.10.007]

Liu Y, Chen X, Ward R K and Wang Z J. 2016. Image fusion with convolutional sparse representation. IEEE Signal Processing Letters, 23(12): 1882-1886 [DOI: 10.1109/LSP.2016.2618776]

Liu Y, Liu S P and Wang Z F. 2015a. A general framework for image fusion based on multi-scale transform and sparse representation. Information Fusion, 24: 147-164 [DOI: 10.1016/j.inffus.2014.09.004]

Liu Y, Liu S P and Wang Z F. 2015b. Multi-focus image fusion with dense SIFT. Information Fusion, 23: 139-155 [DOI: 10.1016/j.inffus.2014.05.004]

Liu Y, Wang L, Cheng J and Chen X. 2021. Multiscale feature interactive network for multifocus image fusion. IEEE Transactions on Instrumentation and Measurement, 70: #5019316 [DOI: 10.1109/TIM.2021.3124058]

Liu Y, Wang L, Cheng J, Li C and Chen X. 2020. Multi-focus image fusion: a survey of the state of the art. Information Fusion, 64: 71-91 [DOI: 10.1016/j.inffus.2020.06.013]

Liu Y and Wang Z F. 2015. Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Processing, 9(5): 347-357 [DOI: 10.1049/iet-ipr.2014.0311]

Liu Z, Blasch E, Xue Z Y, Zhao J Y, Laganiere R and Wu W. 2012. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(1): 94-109 [DOI: 10.1109/TPAMI.2011.109]

Liu Z W, Luo X Q and Zhang Z C. 2020. Multi-focus image fusion with a self-learning fusion rule. Journal of Image and Graphics, 25(8): 1637-1648

刘子闻, 罗晓清, 张战成. 2020. 自学习规则下的多聚焦图像融合. 中国图象图形学报, 25(8): 1637-1648 [DOI: 10.11834/jig.190614]

Ma B Y, Yin X, Wu D, Shen H K, Ban X J and Wang Y. 2022. End-to-end learning for simultaneously generating decision map and multi-focus image fusion result. Neurocomputing, 470: 204-216 [DOI: 10.1016/j.neucom.2021.10.115]

Ma B Y, Zhu Y, Yin X, Ban X J, Huang H Y and Mukeshimana M. 2021a. SESF-Fuse: an unsupervised deep model for multi-focus image fusion. Neural Computing and Applications, 33: 5793-5804 [DOI: 10.1007/s00521-020-05358-9]

Ma H Y, Liao Q M, Zhang JC, Liu S J and Xue J H. 2020. An α-matte boundary defocus model-based cascaded network for multi-focus image fusion. IEEE Transactions on Image Processing, 29: 8668-8679 [DOI: 10.1109/TIP.2020.3018261]

Ma H Y, Zhang J C, Liu S J and Liao Q M. 2019a. Boundary aware multi-focus image fusion using deep neural network//Proceedings of 2019 IEEE International Conference on Multimedia and Expo (ICME). Shanghai, China: IEEE: 1150-1155 [ DOI: 10.1109/ICME.2019.00201 http://dx.doi.org/10.1109/ICME.2019.00201 ]

Ma J L, Zhou Z Q, Wang B, Miao L J and Zong H. 2019b. Multi-focus image fusion using boosted random walks-based algorithm with two-scale focus maps. Neurocomputing, 335: 9-20 [DOI: 10.1016/j.neucom.2019.01.048]

Ma J Y, Le Z L, Tian X and Jiang J J. 2021b. SMFuse: multi-focus image fusion via self-supervised mask-optimization. IEEE Transactions on Computational Imaging, 7: 309-320 [DOI: 10.1109/TCI.2021.3063872]

Mao Q Y, Yang X M, Zhang R Z, Jeon G, Hussain F and Liu K. 2022. Multi-focus images fusion via residual generative adversarial network. Multimedia Tools and Applications, 81(9): 12305-12323 [DOI: 10.1007/s11042-021-11278-0]

Mitianoudis N and Stathaki T. 2007. Pixel-based and region-based image fusion schemes using ICA bases. Information Fusion, 8(2): 131-142 [DOI: 10.1016/j.inffus.2005.09.001]

Murugan A, Arumugam G and Gobinath D. 2021. Multi-focus image fusion using conditional generative adversarial networks//Proceedings of Intelligent Computing and Applications. Singapore, Singapore: Springer: 559-566 [ DOI: 10.1007/978-981-15-5566-4_50 http://dx.doi.org/10.1007/978-981-15-5566-4_50 ]

Mustafa H T, Liu F H, Yang J, Khan Z and Huang Q. 2019a. Dense multi-focus fusion net: a deep unsupervised convolutional network for multi-focus image fusion//Proceedings of the 18th International Conference on Artificial Intelligence and Soft Computing. Zakopane, Poland: Springer: 153-163 [ DOI: 10.1007/978-3-030-20912-4_15 http://dx.doi.org/10.1007/978-3-030-20912-4_15 ]

Mustafa H T, Yang J and Zareapoor M. 2019b. Multi-scale convolutional neural network for multi-focus image fusion. Image and Vision Computing, 85: 26-35 [DOI: 10.1016/j.imavis.2019.03.001]

Mustafa H T, Zareapoor M and Yang J. 2020. MLDNet: multi-level dense network for multi-focus image fusion. Signal Processing: Image Communication, 85: #115864 [DOI: 10.1016/j.image.2020.115864]

Nejati M, Samavi S and Shirani S. 2015. Multi-focus image fusion using dictionary-based sparse representation. Information Fusion, 25: 72-84 [DOI: 10.1016/j.inffus.2014.10.004]

Nian Z C and Jung C. 2019. CNN-based multi-focus image fusion with light field data//Proceedings of 2019 IEEE International Conference on Image Processing (ICIP). Taipei, China: IEEE: 1044-1048 [ DOI: 10.1109/ICIP.2019.8803065 http://dx.doi.org/10.1109/ICIP.2019.8803065 ]

Pan T, Jiang J Q, Yao J, Wang B and Tan B. 2020. A novel multi-focus image fusion network with u-shape structure. Sensors (Basel), 20(14): #3901 [DOI: 10.3390/s20143901]

Peña F A G, Fernández P D M, Ren T I, Vasconcelos G C and Cunha A. 2019. A multiple source hourglass deep network for multi-focus image fusion [EB/OL ] . [2022-05-25 ] . https://arxiv.org/pdf/1908.10945.pdf https://arxiv.org/pdf/1908.10945.pdf

Piella G. 2003. A general framework for multiresolution image fusion: from pixels to regions. Information Fusion, 4(4): 259-280 [DOI: 10.1016/S1566-2535(03)00046-0]

Piella G and Heijmans H. 2003. A new quality metric for image fusion//Proceedings of 2003 International Conference on Image Processing. Barcelona, Spain: IEEE: 173-176 [ DOI: 10.1109/ICIP.2003.1247209 http://dx.doi.org/10.1109/ICIP.2003.1247209 ]

Prabhakar K R, Srikar V S and Babu R V. 2017. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE: 4724-4732 [ DOI: 10.1109/ICCV.2017.505 http://dx.doi.org/10.1109/ICCV.2017.505 ]

Qiu X H, Li M, Zhang L Q and Yuan X J. 2019. Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication, 72: 35-46 [DOI: 10.1016/j.image.2018.12.004]

Song X and Wu X J. 2019. Multi-focus image fusion with PCA filters of PCANet//Proceedings of Multimodal Pattern Recogni tion of Social Signals in Human-Computer-Interaction. Beijing, China: Springer: 1-17 [ DOI: 10.1007/978-3-030-20984-1_1 http://dx.doi.org/10.1007/978-3-030-20984-1_1 ]

Sreedhar P S S and Nandhagopal N. 2022. Classification similarity network model for image fusion using Resnet50 and GoogLeNet. Intelligent Automation and Soft Computing, 31(3): 1331-1344 [DOI: 10.32604/iasc.2022.020918]

Sun J, Zhu H Y, Xu Z B and Han C Z. 2013. Poisson image fusion based on Markov random field fusion model. Information Fusion, 14(3): 241-254 [DOI: 10.1016/j.inffus.2012.07.003]

Tang H, Xiao B, Li W S and Wang G Y. 2018. Pixel convolutional neural network for multi-focus image fusion. Information Sciences, 433-434: 125-141 [DOI: 10.1016/j.ins.2017.12.043]

Wang C, Zhao Z Y, Ren Q Q, Xu Y T and Yu Y. 2020. A novel multi-focus image fusion by combining simplified very deep convolutional networks and patch-based sequential reconstruction strategy. Applied Soft Computing, 91: #106253 [DOI: 10.1016/j.asoc.2020.106253]

Wang C C, Zhou D M, Zang Y S, Nie R C and Guo Y B. 2021a. A deep and supervised atrous convolutional model for multi-focus image fusion. IEEE Sensors Journal, 21(20): 23069-23084 [DOI: 10.1109/JSEN.2021.3106063]

Wang H R, Hua Z and Li J J. 2022. Two-stage progressive residual learning network for multi-focus image fusion. IET Image Processing,16(3): 772-786 [DOI: 10.1049/ipr2.12383]

Wang M, Liu X W and Jin H P. 2019a. A generative image fusion approach based on supervised deep convolution network driven by weighted gradient flow. Image and Vision Computing, 86: 1-16 [DOI: 10.1016/j.imavis.2019.02.011]

Wang Q, Shen Yand Zhang J Q. 2005. A nonlinear correlation measure for multivariable data set. Physica D: Nonlinear Phenomena, 200(3-4): 287-295 [DOI: 10.1016/j.physd.2004.11.001]

Wang Y C, Xu S, Liu J M, Zhao Z X, Zhang C X and Zhang J S. 2021b. MFIF-GAN: a new generative adversarial network for multi-focus image fusion. Signal Processing: Image Communication, 96: #116295 [DOI: 10.1016/j.image.2021.116295]

Wang Z Y, Li X F, Duan H R, Zhang X L and Wang H C. 2019b. Multifocus image fusion using convolutional neural networks in the discrete wavelet transform domain. Multimedia Tools and Applications, 78(24): 34483-34512 [DOI: 10.1007/s11042-019-08070-6]

Wei B Z, Feng X C, Wang K and Gao B. 2021. The multi-focus-image-fusion method based on convolutional neural network and sparse representation. Entropy (Basel), 23(7): #827 [DOI: 10.3390/e23070827]

Wen Y, Yang X M, Celik T, Sushkova O and Albertini M K. 2020. Multifocus image fusion using convolutional neural network. Multimedia Tools and Applications, 79(45): 34531-34543 [DOI: 10.1007/s11042-020-08945-z]

Xiao B, Ou G, Tang H, Bi X L and Li W S. 2020. Multi-focus image fusion by hessian matrix based decomposition. IEEE Transactions on Multimedia, 22(2): 285-297 [DOI: 10.1109/TMM.2019.2928516]

Xiao B, Xu B C, Bi X L and Li W S. 2021. Global-Feature Encoding U-Net (GEU-Net) for multi-focus image fusion. IEEE Transactions on Image Processing, 30: 163-175 [DOI: 10.1109/TIP.2020.3033158]

Xu H, Fan F, Zhang H, Le Z L and Huang J. 2020a. A deep model for multi-focus image fusion based on gradients and connected regions. IEEE Access, 8: 26316-26327 [DOI: 10.1109/ACCESS.2020.2971137]

Xu H, Ma J Y, Jiang J J, Guo X J and Ling H B. 2022. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 502-518 [DOI: 10.1109/TPAMI.2020.3012548]

Xu H, Ma J Y, Le Z L, Jiang J J and Guo X J. 2020b. FusionDN: a unified densely connected network for image fusion. Proceedings of the AAAI Conference on Artificial Intelligence, 34(7): 12484-12491 [DOI: 10.1609/aaai.v34i07.6936]

Xu K P, Qin Z, Wang G L, Zhang H D, Huang K and Ye S X. 2018. Multi-focus image fusion using fully convolutional two-stream network for visual sensors. KSII Transactions on Internet and Information Systems, 12(5): 2253-2272 [DOI: 10.3837/tiis.2018.05.019]

Xu S, Ji L Z, Wang Z, Li P F, Sun K, Zhang C X and Zhang J S. 2020c. Towards reducing severe defocus spread effects for multi-focus image fusion via an optimization based strategy. IEEE Transactions on Computational Imaging, 6: 1561-1570 [DOI: 10.1109/TCI.2020.3039564]

Xu S, Wei X L, Zhang C X, Liu J M and Zhang J S. 2020d. MFFW: a new dataset for multi-focus image fusion [EB/OL ] . [2022-05-25 ] . https://arxiv.org/pdf/2002.04780.pdf https://arxiv.org/pdf/2002.04780.pdf

Xydeas C S and Petrović V. 2000. Objective image fusion performance measure. Electronics Letters, 36(4): 308-309 [DOI: 10.1049/el:20000267]

Yan X, Gilani S Z, Qin H L and Mian A. 2020. Structural similarity loss for learning to fuse multi-focus images. Sensors (Basel), 20(22): #6647 [DOI: 10.3390/s20226647]

Yang B and Li S T. 2010. Multifocus image fusion and restoration with sparse representation. IEEE Transactions on Instrumentation and Measurement, 59(4): 884-892 [DOI: 10.1109/TIM.2009.2026612]

Yang B, Li S T and Sun F M. 2007. Image fusion using nonsubsampled contourlet transform//Proceedings of the 4th International Conference on Image and Graphics (ICIG 2007). Chengdu, China: IEEE: 719-724 [ DOI: 10.1109/ICIG.2007.124 http://dx.doi.org/10.1109/ICIG.2007.124 ]

Yang C, Zhang J Q, Wang X R and Liu X. 2008. A novel similarity based quality metric for image fusion. Information Fusion, 9(2): 156-160 [DOI: 10.1016/j.inffus.2006.09.001]

Yang P, Gao L F and Zi L L. 2021. Image fusion method of convolution sparsity and detail saliency map analysis. Journal of Image and Graphics, 26(10): 2433-2449

杨培, 高雷阜, 訾玲玲. 2021. 卷积稀疏与细节显著图解析的图像融合. 中国图象图形学报, 26(10): 2433-2449 [DOI: 10.11834/jig.200205]

Yang Y, Nie Z P, Huang S Y, Lin P and Wu J H. 2019. Multilevel features convolutional neural network for multifocus image fusion. IEEE Transactions on Computational Imaging, 5(2): 262-273 [DOI: 10.1109/TCI.2018.2889959]

Yin M, Duan P H, Chu B and Liang X Y. 2016. Image fusion based on non-subsampled quaternion shearlet transform. Journal of Image and Graphics, 21(10): 1289-1297

殷明, 段普宏, 褚标, 梁翔宇. 2016. 非下采样四元数剪切波变换域的图像融合. 中国图象图形学报, 21(10): 1289-1297 [DOI: 10.11834/jig.20161003]

Yu N N, Li J J and Hua Z. 2022. Attention based dual path fusion networks for multi-focus image. Multimedia Tools and Applications, 81(8): 10883-10906 [DOI: 10.1007/s11042-022-12046-4]

Zang Y S, Zhou D M, Wang C C, Nie R C and Guo Y B. 2021. UFA-FUSE: a novel deep supervised and hybrid model for multifocus image fusion. IEEE Transactions on Instrumentation and Measurement, 70: #5008717 [DOI: 10.1109/TIM.2021.3072124]

Zeng W H, Li F, Huang H Y, Huang Y and Ding X H. 2019. Two-stream multi-focus image fusion based on the latent decision map//Proceedings of ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton, UK: IEEE: 1762-1766 [ DOI: 10.1109/ICASSP.2019.8683312 http://dx.doi.org/10.1109/ICASSP.2019.8683312 ]

Zhai H and Zhuang Y. 2020. Multi-focus image fusion method using energy of Laplacian and a deep neural network. Applied Optics, 59(6): 1684-1694 [DOI: 10.1364/AO.381082]

Zhang C F. 2022a. Multifocus image fusion using a convolutional elastic network. Multimedia Tools and Applications, 81(1): 1395-1418 [DOI: 10.1007/s11042-021-11362-5]

Zhang H, Le Z L, Shao Z F, Xu H and Ma J Y. 2021. MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Information Fusion, 66: 40-53 [DOI: 10.1016/j.inffus.2020.08.022]

Zhang H, Xu H, Xiao Y, Guo X J and Ma J Y. 2020a. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. Proceedings of the AAAI Conference on Artificial Intelligence, 34(7): 12797-12804 [DOI: 10.1609/aaai.v34i07.6975]

Zhang K, Wu Z L, Yuan X and Zhao C X. 2022. CFNet: context fusion network for multi-focus images. IET Image Processing, 16: 499-508 [DOI: 10.1049/ipr2.12363]

Zhang Q and Guo B L. 2009. Multifocus image fusion using the nonsubsampled contourlet transform. Signal Processing, 89(7): 1334-1346 [DOI: 10.1016/j.sigpro.2009.01.012]

Zhang Q and Levine M D. 2016. Robust multi-focus image fusion using multi-task sparse representation and spatial context. IEEE Transactions on Image Processing, 25(5): 2045-2058 [DOI: 10.1109/TIP.2016.2524212]

Zhang X C. 2022b. Deep learning-based multi-focus image fusion: a survey and a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9): 4819-4838 [DOI: 10.1109/TPAMI.2021.3078906]

Zhang Y, Liu Y, Sun P, Yan H, Zhao X L and Zhang L. 2020b. IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion, 54: 99-118 [DOI: 10.1016/j.inffus.2019.07.011]

Zhao J Y, Laganière R and Liu Z. 2006. Performance assessment of combinative pixel-level image fusion based on an absolute feature measurement. International Journal of Innovative Computing, Information and Control, 3(6A): 1433-1447

Zhao W D, Wang D and Lu H C. 2019. Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 29(4): 1102-1115 [DOI: 10.1109/TCSVT.2018.2821177]

Zhao W Y, Yang H H, Wang J, Pan X P and Cao Z W. 2021. Region-and pixel-level multi-focus image fusion through convolutional neural networks. Mobile Networks and Applications, 26(1): 40-56 [DOI: 10.1007/s11036-020-01719-9]

Zhou D, Jin X, Jiang Q, Cai L, Lee S J and Yao S W. 2022a. MCRD-Net: an unsupervised dense network with multi-scale convolutional block attention for multi-focus image fusion. IET Image Processing, 16(6): 1558-1574 [DOI: 10.1049/ipr2.12430]

Zhou J C, Hao M L, Zhang D H, Zou P Y and Zhang W S. 2019. Fusion PSPnet image segmentation based method for multi-focus image fusion. IEEE Photonics Journal, 11(6): #6501412 [DOI: 10.1109/JPHOT.2019.2950949]

Zhou Y, Liu K, Dou Q Y, Liu Z T, Jeon G and Yang X M. 2022b. LNMF: lightweight network for multi-focus image fusion. Multimedia Tools and Applications, 81(16): 22335-22353 [DOI: 10.1007/s11042-021-11659-5]

Zhou Y, Yang X M, Zhang R Z, Liu K, Anisetti M and Jeon G. 2021. Gradient-based multi-focus image fusion method using convolution neural network. Computers and Electrical Engineering, 92: #107174 [DOI: 10.1016/j.compeleceng.2021.107174]

Zhou Z Q, Li S and Wang B. 2014. Multi-scale weighted gradient-based fusion for multi-focus images. Information Fusion, 20: 60-72 [DOI: 10.1016/j.inffus.2013.11.005]

Zhu Z Q, Yin H P, Chai Y, Li Y X and Qi G Q. 2018. A novel multi-modality image fusion method based on image decomposition and sparse representation. Information Sciences, 432: 516-529 [DOI: 10.1016/j.ins.2017.09.010]