目的 随着深度卷积神经网络的兴起，图像超分重建算法在精度与速度方面均取得长足进展。然而，目前多数超分重建方法需要较深的网络才能取得良好性能，不仅训练难度大，而且到网络末端浅层特征信息容易丢失，难以充分捕获对超分重建起关键作用的高频细节信息。为此，本文融合多尺度特征充分挖掘超分重建所需的高频细节信息，提出了一种全局注意力门控残差记忆网络。方法 在网络前端特征提取部分，利用单层卷积提取浅层特征信息。在网络主体非线性映射部分，级联一组递归的残差记忆模块，每个模块融合多个递归的多尺度残差单元和一个全局注意力门控模块来输出具备多层级信息的特征表征。在网络末端，并联多尺度特征并通过像素重组机制实现高质量的图像放大。结果 本文分别在图像超分重建的5个基准测试数据集（Set5、Set14、B100、Urban100和Manga109）上进行评估，在评估指标峰值信噪比（peak signal to noise ratio，PSNR）和结构相似性（structural similarity，SSIM）上相比当前先进的网络模型均获得更优性能，尤其在Manga109测试数据集上本文算法取得的PSNR结果达到39.19 dB，相比当前先进的轻量型算法AWSRN（adaptive weighted super-resolution network）提高0.32 dB。结论 本文网络模型在对低分图像进行超分重建时，能够联合学习网络多层级、多尺度特征，充分挖掘图像高频信息，获得高质量的重建结果。

单幅图像超分辨率(SISR); 深度卷积神经网络(DCNN); 注意力门控机制; 多尺度残差单元(MRUs); 递归学习

Objective With the rapid development of deep convolutional neural networks (CNNs), great progress has been made in the research of single-image super-resolution (SISR) in terms of accuracy and efficiency. However, existing methods often resort to deeper CNNs that are not only difficult to train, but also have limited feature resolutions to capture the rich high-frequency detail information that is essential for accurate SR prediction. To address these issues, this letter presents a global attention-gated multi-scale memory network (GAMMNet) for SISR. Method GAMMNet mainly consists of three key components: feature extraction, nonlinear mapping (NM), and high-resolution (HR) reconstruction. In the feature extraction part, the input low-resolution (LR) image passes through a 3×3 convolution layer to learn the low-level features. Then, we utilize a recursive structure to perform NM to learn the deeper-layer features. At the end of the network, we arrange four kernels with different sizes followed by a global attention gate (GAG) to achieve the HR reconstruction. Specifically, the NM module consists of four recursive residual memory blocks (RMBs). Each RMB outputs a multi-level representation by fusing the output of the top multi-scale residual unit (MRU) and the ones from the previous MRUs, followed by a GAG module, which serves as a gate to control how much of the previous states should be memorized and decides how much of the current state should be kept. The details of how to design the two novel modules (MRU and GAG) will be explained as follows. MRU takes advantage of the wide activation architecture inspired by the wide activation super-resolution(WDSR) method. As the nonlinear ReLU(rectified linear units) layers in residual blocks hinder the transmission of information flow from shallow layers to deeper layers, the wide activation architecture increases the channels of feature map before ReLU layers to help transmit information flows. Moreover, as the depth of the network increases, problems such as underutilization of features and gradual disappearance of features during transmission occur. Making full use of features is the key to network reconstruction of high-quality images. We parallelly stack two different convolution kernels with sizes of 3×3 and 5×5 to extract multi-scale features, which are further enhanced by the proposed GAG module, yielding the residual output. Finally, the output of MRU is the addition of the input and the residual output. GAG serves as a gate to enhance the input feature channels with different weights. First, different from the residual channel attention network(RCAN) that mainly considers the correlation between feature channels, our GAG takes into account the useful spatial holistic statistic information of the feature maps. We utilize a special spatial pooling operation to improve the global average pooling used in the original channel attention to obtain global context features, and take a weighted average of all location features to establish a more efficient long-range dependency. Then, we aggregate the global context information on each location feature. We first leverage a 1×1 convolution to reduce the feature channel number to 1. Then, we use a Softmax function to capture the global context information for each pixel, yielding the pixel-wise attention weight map. Afterward, we introduce a learning parameter λ₁ to adaptively rescale the weight map. Now the weight map is correlated with the input feature maps, outputting the channel-wise weight vector that encodes the global holistic statistic information of the feature maps. Second, in order to reduce the amount of calculations under the premise of establishing effective long-distance dependencies, we learn to use a bottleneck layer used in SE block to implement feature transform, which not only significantly reduces the amount of calculations but also captures the correlation between feature channels. We feed the channel-wise attention weight vector into a feature transform module that consists of one 1×1 convolution layer, one normalization layer, one ReLU layer, and one 1×1 convolution layer and multiply an adaptively learning parameter λ₂, yielding the enhanced channel-wise attention weights that capture channel-wise dependencies. Finally, we channel-wisely multiply the input feature maps and the enhanced channel-wise attention weights to aggregate the global context information onto each local feature. In the image magnification stage, we design an efficient reconstruction structure to combine local multi-scale features and global features to achieve image magnification. We first leverage three different convolutions followed by a GAG to adaptively adjust the reconstruction feature weights, which make full use of the local multi-scale features of the reconstructed part. Then, a pixel-shuffle module is added behind each branch to perform image magnification. At last, all the reconstructed outputs are added together with the top network branch to combine local and global feature information, outputting the final SR image. Result We adopt the DIV2K(DIVerse 2K resolution image) dataset, the most widely used training dataset for deep CNN-based SISR, to train our model. This dataset contains 1 000 HR images, 800 of which are used for training. We preprocess the HR images by bicubic down-sampling to obtain the LR images. Then, we use several commonly used benchmark datasets, including Set5, Set14, B100, Urban100, and Manga100 to test our model. The evaluation metrics are the peak signal-to-noise ratio (PSNR) and the structural similarity index measurement (SSIM) on the Y channel of the transformed YCbCr space. The input image patches are cropped with a size of 48×48 pixels, and the mini training batch size is set to 16. The hyperparameters of input, internal, and output channel numbers are set to 32, 128, and 32 for the MRUs, respectively. We arrange four RMBs in the non-linear mapping module, among which each block has four MRUs. For the upscale module, we use four different kernel sizes (3×3, 5×5, 7×7, and 9×9), followed by a GAG to generate the HR outputs. We compare our GAMMNet with several state-of-the-art deep CNN-based SISR methods, including super-resolution convolutional neural network(SRCNN), deeply-recursive convolutional network(DRCN), deep recursive residual network(DRRN), memory network(MemNet), cascading residual network(CARN), multi-scale residual network(MSRN), and adaptive weighted super-resolution network(AWSRN). Obviously, our GAMMNet achieves the best performance in terms of both PSNR and SSIM among all compared methods in almost all benchmark datasets, except for the SSIM on the Urban 100, where our GAMMNet achieves the second best performing SSIM of 0.792 6, which is slightly lower than the best AWSRN with SSIM of 0.793 0. Finally, we conduct ablation experiments on the important components of GAMMNet. We use 2×Set5 as the test set. The experiment first replaces the MRUs in GAMMNet with the smallest residual unit in WDSR and removes the GAG as the benchmark. Then, it adds the MRU and GAG and finally trains GAMMNet(simultaneously using the two proposed modules). The results show that the MRU and GAG modules improve PSNR by 0.1 and 0.08 points, respectively, and GAMMNet achieves the best performance on PSNR and SSIM, demonstrating the effectiveness of both module designs. Conclusion In this study, the shallow features of the network are first extracted by the feature extraction module. Then, a nested recursive structure is used to realize the NM and learn the deeper features. This structure combines the features of different scales to effectively learn the context information of the feature maps at each level, and solves the problem of feature disappearing during information transmission by fusing the output of different levels. Finally, in the reconstruction part, the features of different scales are parallelized, and pixel shuffle is used to achieve high-quality magnification of images.

single-image super-resolution(SISR); deep convolutional neural networks(DCNN); attention gate mechanism; multi-scale residual units(MRUs); recursive learning