Current Issue Cover
多尺度渐进式残差网络的图像去雨

卢贝1,2, 盖杉1,2(1. 南昌航空大学信息工程学院, 南昌 330063;2.
2. 江西省图像处理与模式识别重点实验室, 南昌 330063)

摘 要
目的 现有的去雨方法存在去雨不彻底和去雨后图像结构信息丢失等问题。针对这些问题, 提出多尺度渐进式残差网络(multi scale progressive residual network, MSPRNet)的单幅图像去雨方法。方法 提出的多尺度渐进式残差网络通过3个不同感受野的子网络进行逐步去雨。将有雨图像通过具有较大感受野的初步去雨子网络去除图像中的大尺度雨痕。通过残留雨痕去除子网络进一步去除残留的雨痕。将中间去雨结果输入图像恢复子网络, 通过这种渐进式网络逐步恢复去雨过程中损失的图像结构信息。为了充分利用残差网络的残差分支上包含的重要信息, 提出了一种改进残差网络模块, 并在每个子网络中引入注意力机制来指导改进残差网络模块去雨。结果 在5个数据集上与最新的8种方法进行对比实验,相较于其他方法中性能第1的模型, 本文算法在5个数据集上分别获得了0.018、0.028、0.012、0.007和0.07的结构相似度(structural similarity, SSIM)增益。同时在Rain100L数据集上进行了消融实验,实验结果表明,每个子网络的缺失都会造成去雨性能的下降, 提出的多尺度渐进式网络算法能够有效去除各种雨痕。结论 提出的算法能够获得最高的客观评价指标值和最优的视觉效果。在有效解决雨痕重叠问题的同时能够更好地保持图像的细节信息。
关键词
Single image rain removal based on multi scale progressive residual network

Lu Bei1,2, Gai Shan1,2(1. School of Information Engineering, Nanchang Hangkong University, Nanchang 330063, China;2.
2. Key Laboratory of Jiangxi Province for Image Processing and Pattern Recognition, Nanchang 330063, China)

Abstract
Objective Rain streak tends to image degradation, which hinders image restoration, image segmentation and objects orientation. The existing methods are originated from such de-rained methods in related to kernel, low rank approximation and dictionary learning, respectively. The convolutional neural network (CNN) based de-rained image method is commonly facilitated via a single network. It is challenging to remove all rain streaks through a single network due to its multi-shapes, directions and densities.We present a single image rain removal method based on multi scale progressive residual network (MSPRNet). Method A multiscale progressive rain removal network model is proposed based on the residual network. A multi-scale progressive residual network is facilitated, which consists of 3 sub-networks like preliminary rain removal sub network, residual rain streak removal sub network and image restoration sub network. Each sub-network is designed with different receptive fields to process different scale rain streak images and realize gradual rain removal and image restoration. Specifically, the large-scale rain streak of rain image is first removed through the preliminary rain removal sub network with large receptive field to obtain the rain streak image, and then the rain image is obtained. Next, the rain streaks image and the preliminary rain removal image obtained from the previous sub network are as the input, the residual rain streak is further removed through the residual rain streak removal sub network. Finally, the intermediate rain removal image and the residual rain streaks image are input into the image restoration sub network to recover the information of the partial background image removed in the preliminary rain removal sub network. The image structure information loss in the de-rained imaging process is gradually restored through this progressive network. The sparse residual images are derived of rain streaks and some image structures for each sub-network. The residual image is optioned as the intermediate output of the network, and the de-rained image is obtained through subtracting the trained residual rain streaks image in terms of the input of rain image. Each sub network is mainly composed of attention module and improved residual block module. Therefore, the information on the residual branch of the residual block is not fully utilized in the entire network learning process. Considering that each residual branch of residual network contains important information, A newly residual network structure is harnessed in terms of each residual branch of residual network. Simultaneously, attention mechanism is introduced into each sub network to guide the rain removal of the improved residual network module. Meanwhile, the structure of attention module is simplified and guided to reduce the parameters. Result Our experiment is compared with the latest 8 methods on 5 commonly synthetic rain image datasets (Rain100L, Rain100H, Rain12, Rain1200 and Rain1400) and a real rain image dataset. As the training dataset, we selected 200 synthetic rain images from Rain100L, 1 800 synthetic rain images from Rain100H, 1 200 synthetic rain images from Rain1200 and 5 600 synthetic rain images from Rain1400. The demonstrated results are evaluated by peak signal to noise ratio (PSNR), structural similarity (SSIM) and feature similarity (FSIM). The PSNR, SSIM and FSIM obtained MSPRNet are 36.50 dB, 0.985 and 0.988 on the Rain100L dataset, 27.75 dB, 0.895 and 0.931 on the Rain100H dataset, 36.31 dB, 0.971 and 0.979 on the Rain12 dataset, 31.80 dB, 0.932 and 0.963 on the Rain1200 dataset, 32.68 dB, 0.959 and 0.968 on the Rain1200 dataset. The SSIM improvements of MSPRNet over the second best method are 0.018, 0.028, 0.012, 0.007 and 0.07, respectively. In addition, further analysis shows that the performance of MSPRNet on SSIM is better than that of PSNR. SSIM measures image similarity from brightness, contrast and structure, and it is more corresponded to human visual sense than PSNR. In addition, the ablation experiment is conducted based on Rain100L dataset. The experimental results demonstrate that the lack of each sub network will lead to the degradation of rain removal performance. Our multi scale progressive network algorithm can deal with the rain removal effect of a single network. Conclusion Our algorithm can obtain good objective evaluation index value and the qualified visual effect. In the process of rain removal, MSPRNet gradually separates the rain streaks from the rainy images to solve rain streaks overlapping issue effectively. Furthermore, the edge information loss is restored in the process of rain removal through the image restoration sub network. MSPRNet can effectively remove the rain streak and maintain the details of the image.
Keywords

订阅号|日报