A Survey of Hyperspectral Image Super-resolution Method
Jiangtao Nie,Lei Zhang,Wei Wei,Qingsen Yan,Chen Ding,Guochao Chen,Yanning Zhang(Northwestern Polytechnical University;Xi’an University of Posts & Telecommunications)
Different from traditional images like gray and RGB images, hyperspectral images (HSIs) are not only able to provide abundant spatial information but also have discriminative spectral information. Therefore, HSIs are widely applied in various computer vision tasks, including target detection, scene classification, tracking, and so on. However, the limited HSI imaging technology results in the captured HSIs often suffering different degenerations (e.g., low spatial resolution, noise). These degenerations on HSIs damage their image quality and limit their application on subsequent image analysis tasks. Among these degeneration factors, low spatial resolution is a difficult problem that has puzzled researchers for decades years. Recently, numerous HSI super-resolution (SR) methods are proposed to reconstruct high-quality HSIs that not only have a high spectral resolution but also have a high spatial resolution (HR). Existing HSI SR methods can be categorized into spatial SR methods and spectral SR methods. The spatial SR method aims to reconstruct the target HR HSI by improving the spatial resolution of low-resolution (LR) HSI. The spatial SR method can be further divided into single image SR and fusion based SR methods. Single image based SR method reconstructs the target HSI via directly improving the spatial resolution of LR HSI. However, due to lost too much spatial information, the single image based SR method hard to reconstruct satisfying HSIs. Therefore, some researchers propose to introduce extra high spatial information from other images (e.g., multispectral image (MSI), RGB) that have the same imaging scene information with the HSI, and fuse these high quality spatial information into the LR HSIs. In this way, the spatial resolution of LR HSI can be improved greatly (e.g., 8 times, 16 times, and 32 times SR). The other kind of HSI SR method is the spectral super-resolution method, which aims to directly improve the spectral resolution of high spatial resolution images (e.g., multispectral image, RGB) and generate the target hyperspectral image. In this study, we successively introduce the development of HSI SR from the aspects of single image HSI SR method, fusion based HSI SR method, and spectral super-resolution method. Each of these three category HSI SR methods can be further divided into traditional optimization framework based methods and deep learning based methods. For single image HSI SR, due to the SR problem being an illness inverse problem, the aim of traditional optimization framework based single image HSI SR methods is to exploit effective image priors to restrain the SR process. Among the image priors, low-rank, sparse representation, and non-local are some common priors that are adopted by most single image based HSI SR methods. However, these image priors are heuristic handcrafted and have limited restriction ability, this may damage the application of these methods in challenging cases. For the traditional fusion based method, the success is lie in how to jointly exploit the spatial-spectral correlation between HR MSI and LR HSI. A promising way is to decompose these two images into key components and then re-combine the useful parts of each image to form the reconstructed HR HSI. Therefore, kinds of decomposition schemes are introduced (e.g., non-negative matrix factorization, coupled tensor factorization) to separate the key information from HR MSI and LR HSI. In addition, to increase the effectiveness of these decomposition methods, some constraints will be introduced (e.g., sparse, low-rank). However, this kind of method often has limited representation ability and may fail to exploit the inherent correlation between HR MSI and LR HSI when the SR problem is challenging. For the traditional spectral resolution method, it is essential to learn how to reconstruct the spectral characteristics from RGB/MSI images. When paired RGB and HSI exist, a promising way is to construct a dictionary (e.g., sparse dictionary learning) that records the mapping relation between RGB/MSI images and HSIs. The dictionary learning based spectral super-resolution methods have achieved great success, but this kind of method often has limited generalization ability, which may restrict their application situations. In recent years, deep learning based methods have achieved great success on most computer vision tasks. The powerful feature representation ability benefits the deep learning based methods to exploit deeper characteristics from images that benefit various vision tasks, and it also empowers us to explore the inherent spatial-spectral relations of HSIs. For the single image HSI SR methods, utilizing a deep convolution neural network (DCNN) to learn the mapping process from LR HSI to HR HSI is a promising manner. In this way, the DCNN is able to learn deep image prior from plenty of training samples, which has better representation ability than the heuristic handcrafted image priors (e.g., low-rank, sparse) to some extent. However, the performance of this kind of method is often restricted by the amount and variety of training samples. Therefore, some researchers propose to reconstruct HR HSI from LR HSI via an unsupervised manner, but how to improve the robustness of the unsupervised method still is a challenging problem. For the deep learning based HSI fusion methods, researchers often focus on designing effective DCNN frameworks (e.g., multi-branch, multi-scale, 3D-CNN) to extract spatial-spectral information from MSIs and HSIs. However, these also exist several challenging problems (e.g., noise, unknown degeneration, unregistered) that restrict the generalization ability of the existing DCNN based HSI fusion method. To address these problems, kinds of technologies are introduced, in which unsupervised learning and adaptation learning are introduced to improve the generalization ability of fusion based method, DCNN unfolding is proposed to improve the interpretability, and registration strategy is introduced to improve the robustness. For the spectral super resolution method, DCNN brings a powerful manner to model the mapping scheme from RGB/MSI to HR HSI, and leads to obvious performance improvement. However, there still have some challenging problems that impede the development of DCNN based spectral super resolution method. For example, most existing spectral super resolution methods are only able to generate HSI with fixed spectral interval or fixed spectrum range. Thus, how to construct a spectral super resolution framework that has well generalization ability is a scope that worthy of study. In summary, we divide the HSI SR method into three categories and introduce each category in the main manuscript in detail. We organize this study from the perspective of new designs, new methods, and new application scenes of the HSI SR scope, and put the focus on the state-of-the-art method, hot issues, and new trends.
hyperspectral image super-resolution single image super-resolution fusion based super-resolution spectral super-resolution