Current Issue Cover
高光谱图像超分辨率重建技术研究进展

聂江涛,张磊,魏巍,闫庆森,丁晨,陈国超,张艳宁(西北工业大学;西安邮电大学)

摘 要
不同于传统图像 (如灰度图像、RGB图像等)专注于保存目标场景的空间信息,高光谱图像蕴含丰富的空间-光谱信息,其不仅可以保存目标的空间信息,同时还可以保存具有高可辨性的光谱信息。因此高光谱图像被应用于多种计算机视觉和遥感图像任务中,如目标检测、场景分类和目标追踪等。然而,在高光谱图像获取以及重建过程中仍然存在着许多问题与瓶颈。如传统高光谱成像仪器在成像过程中通常会引入噪声,且获得的图像往往具有较低的空间分辨率,这极大的影响了高光谱图像的质量,对后续数据分析任务造成了极大的困难。近年来,高光谱图像超分辨率重建技术研究得到了极大的发展,现有超分辨率重建方法可以大致分为两类,一类为空间超分辨率重建方法,可以直接提升高光谱图像的空间分辨率来获得高质量高光谱图像。另一类为光谱超分辨率重建方法,可以通过提升高空间分辨率图像的光谱分辨率来生成高质量高光谱图像。本选题拟从高光谱图像超分辨率重建领域的新设计、新方法和应用场景出发,通过综合国内外前沿文献来梳理该领域的主要发展。重点论述高光谱图像超分辨率重建领域的发展现状、前沿动态、热点问题及趋势。
关键词
A Survey of Hyperspectral Image Super-resolution Method

Jiangtao Nie,Lei Zhang,Wei Wei,Qingsen Yan,Chen Ding,Guochao Chen,Yanning Zhang(Northwestern Polytechnical University;Xi’an University of Posts & Telecommunications)

Abstract
Different from traditional images like gray and RGB images, hyperspectral images (HSIs) are not only able to provide abundant spatial information but also have discriminative spectral information. Therefore, HSIs are widely applied in various computer vision tasks, including target detection, scene classification, tracking, and so on. However, the limited HSI imaging technology results in the captured HSIs often suffering different degenerations (e.g., low spatial resolution, noise). These degenerations on HSIs damage their image quality and limit their application on subsequent image analysis tasks. Among these degeneration factors, low spatial resolution is a difficult problem that has puzzled researchers for decades years. Recently, numerous HSI super-resolution (SR) methods are proposed to reconstruct high-quality HSIs that not only have a high spectral resolution but also have a high spatial resolution (HR). Existing HSI SR methods can be categorized into spatial SR methods and spectral SR methods. The spatial SR method aims to reconstruct the target HR HSI by improving the spatial resolution of low-resolution (LR) HSI. The spatial SR method can be further divided into single image SR and fusion based SR methods. Single image based SR method reconstructs the target HSI via directly improving the spatial resolution of LR HSI. However, due to lost too much spatial information, the single image based SR method hard to reconstruct satisfying HSIs. Therefore, some researchers propose to introduce extra high spatial information from other images (e.g., multispectral image (MSI), RGB) that have the same imaging scene information with the HSI, and fuse these high quality spatial information into the LR HSIs. In this way, the spatial resolution of LR HSI can be improved greatly (e.g., 8 times, 16 times, and 32 times SR). The other kind of HSI SR method is the spectral super-resolution method, which aims to directly improve the spectral resolution of high spatial resolution images (e.g., multispectral image, RGB) and generate the target hyperspectral image. In this study, we successively introduce the development of HSI SR from the aspects of single image HSI SR method, fusion based HSI SR method, and spectral super-resolution method. Each of these three category HSI SR methods can be further divided into traditional optimization framework based methods and deep learning based methods. For single image HSI SR, due to the SR problem being an illness inverse problem, the aim of traditional optimization framework based single image HSI SR methods is to exploit effective image priors to restrain the SR process. Among the image priors, low-rank, sparse representation, and non-local are some common priors that are adopted by most single image based HSI SR methods. However, these image priors are heuristic handcrafted and have limited restriction ability, this may damage the application of these methods in challenging cases. For the traditional fusion based method, the success is lie in how to jointly exploit the spatial-spectral correlation between HR MSI and LR HSI. A promising way is to decompose these two images into key components and then re-combine the useful parts of each image to form the reconstructed HR HSI. Therefore, kinds of decomposition schemes are introduced (e.g., non-negative matrix factorization, coupled tensor factorization) to separate the key information from HR MSI and LR HSI. In addition, to increase the effectiveness of these decomposition methods, some constraints will be introduced (e.g., sparse, low-rank). However, this kind of method often has limited representation ability and may fail to exploit the inherent correlation between HR MSI and LR HSI when the SR problem is challenging. For the traditional spectral resolution method, it is essential to learn how to reconstruct the spectral characteristics from RGB/MSI images. When paired RGB and HSI exist, a promising way is to construct a dictionary (e.g., sparse dictionary learning) that records the mapping relation between RGB/MSI images and HSIs. The dictionary learning based spectral super-resolution methods have achieved great success, but this kind of method often has limited generalization ability, which may restrict their application situations. In recent years, deep learning based methods have achieved great success on most computer vision tasks. The powerful feature representation ability benefits the deep learning based methods to exploit deeper characteristics from images that benefit various vision tasks, and it also empowers us to explore the inherent spatial-spectral relations of HSIs. For the single image HSI SR methods, utilizing a deep convolution neural network (DCNN) to learn the mapping process from LR HSI to HR HSI is a promising manner. In this way, the DCNN is able to learn deep image prior from plenty of training samples, which has better representation ability than the heuristic handcrafted image priors (e.g., low-rank, sparse) to some extent. However, the performance of this kind of method is often restricted by the amount and variety of training samples. Therefore, some researchers propose to reconstruct HR HSI from LR HSI via an unsupervised manner, but how to improve the robustness of the unsupervised method still is a challenging problem. For the deep learning based HSI fusion methods, researchers often focus on designing effective DCNN frameworks (e.g., multi-branch, multi-scale, 3D-CNN) to extract spatial-spectral information from MSIs and HSIs. However, these also exist several challenging problems (e.g., noise, unknown degeneration, unregistered) that restrict the generalization ability of the existing DCNN based HSI fusion method. To address these problems, kinds of technologies are introduced, in which unsupervised learning and adaptation learning are introduced to improve the generalization ability of fusion based method, DCNN unfolding is proposed to improve the interpretability, and registration strategy is introduced to improve the robustness. For the spectral super resolution method, DCNN brings a powerful manner to model the mapping scheme from RGB/MSI to HR HSI, and leads to obvious performance improvement. However, there still have some challenging problems that impede the development of DCNN based spectral super resolution method. For example, most existing spectral super resolution methods are only able to generate HSI with fixed spectral interval or fixed spectrum range. Thus, how to construct a spectral super resolution framework that has well generalization ability is a scope that worthy of study. In summary, we divide the HSI SR method into three categories and introduce each category in the main manuscript in detail. We organize this study from the perspective of new designs, new methods, and new application scenes of the HSI SR scope, and put the focus on the state-of-the-art method, hot issues, and new trends.
Keywords

订阅号|日报