光谱图像超分辨率技术的目标是从低空间分辨率和光谱分辨率的图像中恢复高空间分辨率和光谱分辨率的图像,并在植被调查、地质勘探、环境保护、异常检测、目标追踪等遥感领域有着广泛应用。随着深度学习的兴起,基于深度学习的光谱图像超分辨率算法如雨后春笋般涌现,特别是卷积神经网络、Transformer、生成对抗网络以及深度展开网络等技术的出现使图像融合性能产生了质的飞跃。本文全面探讨和分析了不同光谱图像超分辨率任务场景下的前沿深度学习算法。首先,本文介绍了光谱图像超分辨率的基本概念,以及不同超分辨率场景的定义。针对单图超分辨率和融合超分辨率两大场景,从超分辨率维度、数据类型、基本框架、监督方式等多个角度详细阐述了各类方法的基本思想和特点。其次,本文总结了各类算法存在的局限性,并提出了进一步改进的方向。进一步,简要介绍了不同融合场景中常用的数据集,并明确了各种评估指标的具体定义。对于每种超分辨率任务,本文从定性评估、定量评估等多个角度全面比较了代表性算法的性能。本文提及的算法和数据集已汇总至https://github.com/ColinTaoZhang/DL-based-spectral-super-resolution。最后,本文总结了研究结果,并探讨了光谱图像超分辨率领域所面临的一些严峻挑战,同时对未来可能的研究方向进行了展望。

The goal of spectral image super-resolution technology is to recover images with high spatial resolution and spectral resolution from images with low spatial resolution and spectral resolution. The high spatial and spectral resolution image is widely used in remote sensing fields such as vegetation survey, geological exploration, environmental protection, anomaly detection and target tracking. With the rise of deep learning, spectral image super-resolution algorithms based on deep learning have sprung up. In particular, the emergence of technologies such as end-to-end neural networks, generative adversarial networks, and deep unfolding networks has made a qualitative leap in spectral image super-resolution performance. This article comprehensively discusses and analyzes cutting-edge deep learning algorithms under different spectral image super-resolution task scenarios. First, we introduce the basic concepts of spectral image super-resolution and the definitions of different super-resolution scenarios. Focusing on the two major scenarios of single-image super-resolution and fusion super-resolution, the basic concepts of various methods are elaborated from multiple perspectives such as super-resolution dimensions, super-resolution data types, basic frameworks, and supervision methods, and their characteristics are discussed. Secondly, this article summarizes the limitations of various algorithms and proposes directions for further improvement. Furthermore, the commonly used data sets in different fusion scenarios are briefly introduced, and the specific definitions of various evaluation indicators are clarified. For each super-resolution task, this paper comprehensively compares the performance of representative algorithms from multiple perspectives such as qualitative evaluation and quantitative evaluation. Finally, this article summarizes the research results and discusses some serious challenges faced in the field of spectral image super-resolution, while also looking forward to possible future research directions. First of all, from the perspective of super-resolution scenarios, the existing spectral image super-resolution algorithms can be divided into two categories, including single image super-resolution and fusion-based super-resolution. Specifically, single spectral image super-resolution is designed to generate high-resolution output images from a single low-resolution input image. According to the direction of super-resolution, single image super-resolution can be divided into spatial super-resolution, spectral super-resolution and spatio-spectral super-resolution. Fusion-based spectral image super-resolution is designed to fuse images of different modes into a single image with high spatial and spectral resolution. According to the different modes of fusion images, fusion-based spectral image super-resolution can be divided into pansharpening and multispectral and hyperspectral images fusion. Moreover, deep learning-based spectral image super-resolution methods can be categorized into end-to-end neural network based (E2EN-based) spectral image super-resolution framework, generative adversarial network based (GAN-based) spectral image super-resolution framework, and deep unfolding network based (DUN-based) spectral image super-resolution framework according to the network architecture. The E2EN-based spectral image super-resolution framework designs various network structures to mine nonlinear mapping relationships between low-resolution and high-resolution images. According to the basic computing unit of network structure, it can be divided into convolutional neural network (CNN) based method and Transformer based method. The GAN-based spectral image super resolution framework realizes the spectral image super resolution through the game between generator and discriminator. The DUN-based spectral image super-resolution framework combines traditional optimization algorithms and deep learning, and unfolds iterative optimization steps to form deep neural networks. From the perspective of supervision paradigm, the deep learning algorithms can also be classified into unsupervised and supervised categories. The supervised approaches minimize the distance between super-resolved spectral image and ground truth, while unsupervised algorithms design loss function through the similarity between super-resolved and input images or through the game of generator and discriminator. Our critical review describes the main concepts and characteristics of each approach for different spectral image super-resolution tasks according to the network architecture and supervision paradigm. Secondly, we introduce the representative datasets and evaluation metrics. We divide the datasets into single spectral image super-resolution datasets and fusion-based spectral image super-resolution datasets categories. Besides, the evaluation metrics can be grouped into full-reference metrics and no-reference metrics. There some full-reference metrics are widely used for the quantitative evaluation of spectral image super-resolution, including peak signal-to-noise (PSNR), structural similarity (SSIM), spectral angle mapper (SAM) and relative dimensionless global error in synthesis (ERGAS). Thirdly, we provide the quantitative and qualitative experimental results of different spectral image super-resolution tasks. Finally, we summarize the challenges problems in the study of deep learning-based spectral image super-resolution, and carried out forecasting analysis, such as high-quality spectral image super-resolution dataset, model-driven and deep learning combined spectral image super-resolution method, real-time spectral image super-resolution and comprehensive evaluation metrics. The methods and datasets mentioned are linked at: https://github.com/ColinTaoZhang/DL-based-spectral-super-resolution.