Current Issue Cover
自适应卷积的残差修正单幅图像去雨

王美华, 何海君, 李超(华南农业大学数学与信息学院, 广州 510642)

摘 要
目的 雨天户外采集的图像常常因为雨线覆盖图像信息产生色变和模糊现象。为了提高雨天图像的质量,本文提出一种基于自适应选择卷积网络深度学习的单幅图像去雨算法。方法 针对雨图中背景误判和雨痕残留问题,加入网络训练的雨线修正系数(refine factor,RF),改进现有雨图模型,更精确地描述雨图中各像素受到雨线的影响。构建选择卷积网络(selective kernel network,SK Net),自适应地选择不同卷积核对应维度的信息,进一步学习、融合不同卷积核的信息,提高网络的表达力,最后构建包含SK Net、refine factor net和residual net子网络的自适应卷积残差修正网络(selective kernel convolution using a residual refine factor,SKRF),直接学习雨线图和残差修正系数(RF),减少映射区间,减少背景误判。结果 实验通过设计的SKRF网络,在公开的Rain12测试集上进行去雨实验,取得了比现有方法更高的精确度,峰值信噪比(peak signal to noise ratio,PSNR)达到34.62 dB,结构相似性(structural similarity,SSIM)达到0.970 6。表明SKRF网络对单幅图像去雨效果有明显优势。结论 单幅图像去雨SKRF算法为雨图模型中的雨线图提供一个额外的修正残差系数,以降低学习映射区间,自适应选择卷积网络模型提升雨图模型的表达力和兼容性。
关键词
Single image rain removal based on selective kernel convolution using a residual refine factor

Wang Meihua, He Haijun, Li Chao(College of Mathematics and Informatics, South China Agricultural University, Guangzhou 510642, China)

Abstract
Objective Rain lines adversely affect the visual quality of images collected from outdoors. Severe weather conditions, such as rain, fog, and haze, can affect the quality of these images and make them unusable. These degraded images may also drastically affect the performance of man's vision system. Given that rain is a common meteorological phenomenon, an algorithm that can remove rain from single image is of practical significance. Given that video-based de-raining methods obtain pixel information of the same location at different periods, removing rain from an individual image is more challenging because of less available information. Traditional de-raining methods mainly focus on rain map modeling and use mathematical optimization to detect and remove rain streaks, but the performance of such approach requires further improvement. Method To address the above problems, this paper establishes a convolution neural network for single image rain removal that is trained on a synthetic dataset. The contributions of this work are as follows. 1) To expand the neural receptive field of a convolution neural network that learns abstract feature representation of rain streaks and the ground truth, this work establishes a selective kernel network based on multi-scale convolution with different kernel for feature learning. To accomplish useful information fusion and selection, an external non-linear weight learning mechanism is developed to redistribute the weight for the corresponding channel's feature information from different convolution kernels. This mechanism enables the network to select the feature information of different receptive fields adaptively and enhance its expression ability and rain removal capability. 2) The existing rain map model shows some limitations at the training stage. Completing this model by adding a learnable refine factor that modifies each pixel in a rain streak image, can enhance the accuracy of the result and prevent background misjudgment. The range of the refining factor is also limited to reduce the mapping range of the network training process. 3) At the training stage the existing single image rain removal networks need to learn various types of image content, including rain streaks removal and background restoration, which will undoubtedly increase their burden. By using the novel idea of residual learning the proposed network can directly learn the rain streak map by using the input rain map. In this way, the mapping interval of the network learning process is reduced, the background of the original graph can be preserved, and loss of details can be prevented. The validity of the above arguments is tested by designing a comparison network with different modules. Specifically, based on general convolution, different modules are combined step by step, including the SK net, residual learning mechanism, and refine factor learning net. Single image rain removal network based on selective kernel convolution using residual refine factor (SKRF) is eventually designed. The residual learning mechanism is used to reduce the mapping interval, and the refined factor is used to enhance the rain streak map to improve the rain removal performance. Result An SKRF network, including the three subnets of SK net, refine factor net, and residual net, is designed in a rain removal experiment and tested on the open Rain12 test set. This network achieves a higher accuracy, peak signal to noise ratio(PSNR) (34.62), and structural similarity(SSIM) (0.970 6) compared with the existing methods. The SKRF network shows obvious advantages in removing rain from single image. Conclusion We construct a convolution neural network based on SKRF to remove rain streaks from single image. A selective kernel convolution network is established to improve the expression ability of the proposed network via the adaptive adjustment mechanism of the size of the receptive field by the internal neurons. A rain map with different characteristics can be well learned, and the effect of rain removal can be improved. The residual learning mechanism can reduce the mapping interval of the network learning process and retain more details of the original image. In the modified rain map model, an additional refine factor is provided for the rain streak map, which can further reduce the mapping interval and reduce background misjudgment. This network not only removes the majority of the visible rain streaks but also retains the ground truth. In our feature work, we plan to extend this network to a wider range of image restoration tasks.
Keywords

订阅号|日报