浏览全部资源
扫码关注微信
纸质出版日期： 2022-05-16 ，
录用日期： 2021-10-26
移动端阅览
卢贝, 盖杉. 多尺度渐进式残差网络的图像去雨[J]. 中国图象图形学报, 2022,27(5):1537-1553.
Bei Lu, Shan Gai. Single image rain removal based on multi scale progressive residual network[J]. Journal of Image and Graphics, 2022,27(5):1537-1553.
目的
2
现有的去雨方法存在去雨不彻底和去雨后图像结构信息丢失等问题。针对这些问题
提出多尺度渐进式残差网络(multi scale progressive residual network
MSPRNet)的单幅图像去雨方法。
方法
2
提出的多尺度渐进式残差网络通过3个不同感受野的子网络进行逐步去雨。将有雨图像通过具有较大感受野的初步去雨子网络去除图像中的大尺度雨痕。通过残留雨痕去除子网络进一步去除残留的雨痕。将中间去雨结果输入图像恢复子网络
通过这种渐进式网络逐步恢复去雨过程中损失的图像结构信息。为了充分利用残差网络的残差分支上包含的重要信息
提出了一种改进残差网络模块
并在每个子网络中引入注意力机制来指导改进残差网络模块去雨。
结果
2
在5个数据集上与最新的8种方法进行对比实验，相较于其他方法中性能第1的模型
本文算法在5个数据集上分别获得了0.018、0.028、0.012、0.007和0.07的结构相似度(structural similarity
SSIM)增益。同时在Rain100L数据集上进行了消融实验，实验结果表明，每个子网络的缺失都会造成去雨性能的下降
提出的多尺度渐进式网络算法能够有效去除各种雨痕。
结论
2
提出的算法能够获得最高的客观评价指标值和最优的视觉效果。在有效解决雨痕重叠问题的同时能够更好地保持图像的细节信息。
Objective
2
Rain streak tends to image degradation
which hinders image restoration
image segmentation and objects orientation. The existing methods are originated from such de-rained methods in related to kernel
low rank approximation and dictionary learning
respectively. The convolutional neural network (CNN) based de-rained image method is commonly facilitated via a single network. It is challenging to remove all rain streaks through a single network due to its multi-shapes
directions and densities.We present a single image rain removal method based on multi scale progressive residual network (MSPRNet).
Method
2
A multiscale progressive rain removal network model is proposed based on the residual network. A multi-scale progressive residual network is facilitated
which consists of 3 sub-networks like preliminary rain removal sub network
residual rain streak removal sub network and image restoration sub network. Each sub-network is designed with different receptive fields to process different scale rain streak images and realize gradual rain removal and image restoration. Specifically
the large-scale rain streak of rain image is first removed through the preliminary rain removal sub network with large receptive field to obtain the rain streak image
and then the rain image is obtained. Next
the rain streaks image and the preliminary rain removal image obtained from the previous sub network are as the input
the residual rain streak is further removed through the residual rain streak removal sub network. Finally
the intermediate rain removal image and the residual rain streaks image are input into the image restoration sub network to recover the information of the partial background image removed in the preliminary rain removal sub network. The image structure information loss in the de-rained imaging process is gradually restored through this progressive network. The sparse residual images are derived of rain streaks and some image structures for each sub-network. The residual image is optioned as the intermediate output of the network
and the de-rained image is obtained through subtracting the trained residual rain streaks image in terms of the input of rain image. Each sub network is mainly composed of attention module and improved residual block module. Therefore
the information on the residual branch of the residual block is not fully utilized in the entire network learning process. Considering that each residual branch of residual network contains important information
A newly residual network structure is harnessed in terms of each residual branch of residual network. Simultaneously
attention mechanism is introduced into each sub network to guide the rain removal of the improved residual network module. Meanwhile
the structure of attention module is simplified and guided to reduce the parameters.
Result
2
Our experiment is compared with the latest 8 methods on 5 commonly synthetic rain image datasets (Rain100L
Rain100H
Rain12
Rain1200 and Rain1400) and a real rain image dataset. As the training dataset
we selected 200 synthetic rain images from Rain100L
1 800 synthetic rain images from Rain100H
1 200 synthetic rain images from Rain1200 and 5 600 synthetic rain images from Rain1400. The demonstrated results are evaluated by peak signal to noise ratio (PSNR)
structural similarity (SSIM) and feature similarity (FSIM). The PSNR
SSIM and FSIM obtained MSPRNet are 36.50 dB
0.985 and 0.988 on the Rain100L dataset
27.75 dB
0.895 and 0.931 on the Rain100H dataset
36.31 dB
0.971 and 0.979 on the Rain12 dataset
31.80 dB
0.932 and 0.963 on the Rain1200 dataset
32.68 dB
0.959 and 0.968 on the Rain1200 dataset. The SSIM improvements of MSPRNet over the second best method are 0.018
0.028
0.012
0.007 and 0.07
respectively. In addition
further analysis shows that the performance of MSPRNet on SSIM is better than that of PSNR. SSIM measures image similarity from brightness
contrast and structure
and it is more corresponded to human visual sense than PSNR. In addition
the ablation experiment is conducted based on Rain100L dataset. The experimental results demonstrate that the lack of each sub network will lead to the degradation of rain removal performance. Our multi scale progressive network algorithm can deal with the rain removal effect of a single network.
Conclusion
2
Our algorithm can obtain good objective evaluation index value and the qualified visual effect. In the process of rain removal
MSPRNet gradually separates the rain streaks from the rainy images to solve rain streaks overlapping issue effectively. Furthermore
the edge information loss is restored in the process of rain removal through the image restoration sub network. MSPRNet can effectively remove the rain streak and maintain the details of the image.
单幅图像去雨深度学习卷积神经网络(CNN)残差网络注意力机制
single image de-raineddeep learningconvolutional neural network(CNN)residual networkattention mechanism
Bi X J and Xing J Y. 2020. Multi-scale weighted fusion attentive generative adversarial network for single image de-raining. IEEE Access, 8: 69838-69848 [DOI: 10.1109/ACCESS.2020.2983436]
Chang Y, Yan L X and Zhong S. 2017. Transformed low-rank model for line pattern noise removal//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 1735-1743 [DOI: 10.1109/ICCV.2017.191http://dx.doi.org/10.1109/ICCV.2017.191]
Chen L, Zhang H and Xiao J. 2017. SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 6298-6306 [DOI: 10.1109/CVPR.2017.667http://dx.doi.org/10.1109/CVPR.2017.667]
Chen Q W, Xie H W, Zha H, Xi Y and Zhang X. 2021. Salient object detection based on deep clustering attention mechanism. Journal of Image and Graphics, 26(5): 1017-1029
陈庆文, 谢宏文, 查浩, 奚瑜, 张雪. 2021. 深度聚类注意力机制下的显著对象检测. 中国图象图形学报, 26(5): 1017-1029 [DOI: 10.11834/jig.200081]
Chen Y L and Hsu C T. 2013. A generalized low-rank appearance model for spatio-temporally correlated rain streaks//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, Australia: IEEE: 1968-1975 [DOI: 10.1109/ICCV.2013.247http://dx.doi.org/10.1109/ICCV.2013.247]
Cheng D Q, Guo X, Chen L L, Kou Q Q, Zhao K and Gao R. 2021. Image super-resolution reconstruction from multi-channel recursive residual network. Journal of Image and Graphics, 26(3): 605-618
程德强, 郭昕, 陈亮亮, 寇旗旗, 赵凯, 高蕊. 2021. 多通道递归残差网络的图像超分辨率重建. 中国图象图形学报, 26(3): 605-618 [DOI: 10.11834/jig.200108]
Fu X Y, Huang J B, Ding X H, Liao Y H and Paisley J. 2017a. Clearing the skies: a deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6): 2944-2956 [DOI: 10.1109/TIP.2017.2691802]
Fu X Y, Huang J B, Zeng D L, Huang Y, Ding X H and Paisley J. 2017b. Removing rain from single images via a deep detail network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 1715-1723 [DOI: 10.1109/CVPR.2017.186http://dx.doi.org/10.1109/CVPR.2017.186]
Fu X Y, Liang B R, Huang Y, Ding X H and Paisley J. 2020. Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems, 31(6): 1794-1807 [DOI: 10.1109/TNNLS.2019.2926481]
Hu J, Shen L, Albanie S, Sun G and Wu E H. 2020. Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8): 2011-2023 [DOI: 10.1109/TPAMI.2019.2913372]
Jiang K, Wang Z Y, Yi P, Chen C, Huang B J, Luo Y M, Ma J Y and Jiang J J. 2020. Multi-scale progressive fusion network for single image deraining//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 8343-8352 [DOI: 10.1109/CVPR42600.2020.00837http://dx.doi.org/10.1109/CVPR42600.2020.00837]
Kang L W, Lin C W and Fu Y H. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, 21(4): 1742-1755 [DOI: 10.1109/TIP.2011.2179057]
Kim J H, Lee C, Sim J Y and Kim C S. 2013. Single-image deraining using an adaptive nonlocal means filter//Proceedings of 2013 IEEE International Conference on Image Processing. Melbourne, Australia: IEEE: 914-917 [DOI: 10.1109/ICIP.2013.6738189http://dx.doi.org/10.1109/ICIP.2013.6738189]
Kim J H, Sim J Y and Kim C S. 2015. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing, 24(9): 2658-2670 [DOI: 10.1109/TIP.2015.2428933]
Kingma D P and Ba J. 2015. Adam: a method for stochastic optimization//Proceedings of the 3rd International Conference on Learning Representations. San Diego, USA: [s. n.]: 273-297
Li Y, Tan R T, Guo X J, Lu J B and Brown M S. 2016. Rain streak removal using layer priors//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 2736-2744 [DOI: 10.1109/CVPR.2016.299http://dx.doi.org/10.1109/CVPR.2016.299]
Lin T Y, Goyal P, Girshick R, He K M and Dollár P. 2017. Focal loss for dense object detection//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 2999-3007 [DOI: 10.1109/ICCV.2017.324http://dx.doi.org/10.1109/ICCV.2017.324].
Lu C Y, Tang J H, Yan S C and Lin Z C. 2014. Generalized nonconvex nonsmooth low-rank minimization//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 4130-4137 [DOI: 10.1109/CVPR.2014.526http://dx.doi.org/10.1109/CVPR.2014.526]
Luo Y, Xu Y and JiH. 2015. Removing rain from a single image via discriminative sparse coding//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 3397-3405 [DOI: 10.1109/ICCV.2015.388http://dx.doi.org/10.1109/ICCV.2015.388]
Peng J Y, Xu Y, Chen T Y and Huang Y. 2020. Single-image raindrop removal using concurrent channel-spatial attention and long-short skip connections. Pattern Recognition Letters, 131: 121-127 [DOI: 10.1016/j.patrec.2019.12.012]
Qian R, Tan R T, Yang W H, Su J J and Liu J Y. 2018. Attentive generative adversarial network for raindrop removal from a single image//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2482-2491 [DOI: 10.1109/CVPR.2018.00263http://dx.doi.org/10.1109/CVPR.2018.00263]
Wang T Y, Yang X, Xu K, Chen S Z, Zhang Q and Lau R W H. 2019. Spatial attentive single-image deraining with a high quality real rain dataset//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 12262-12271 [DOI: 10.1109/CVPR.2019.01255http://dx.doi.org/10.1109/CVPR.2019.01255]
Wang Y L, Liu S C, Chen C and Zeng B. 2017. A hierarchical approach for rain or snow removing in a single color image. IEEE Transactions on Image Processing, 26(8): 3936-3950 [DOI: 10.1109/TIP.2017.2708502]
Wang Y T, Zhao X L, Jiang T X, Deng L J, Chang Y and Huang T Z. 2021. Rain streaks removal for single image via kernel-guided convolutional neural network. IEEE Transactions on Neural Networks and Learning Systems, 32(8): 3664-3676 [DOI: 10.1109/TNNLS.2020.3015897]
Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612 [DOI: 10.1109/TIP.2003.819861]
Woo S, Park J, Lee J Y and Kweon I S. 2018. CBAM: convolutional block attention module//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 3-19 [DOI: 10.1007/978-3-030-01234-2_1http://dx.doi.org/10.1007/978-3-030-01234-2_1]
Yang W H, Tan R T, Feng J S, Liu J Y, Guo Z M and Yan S C. 2017. Deep joint rain detection and removal from a single image//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 1685-1694 [DOI: 10.1109/CVPR.2017.183http://dx.doi.org/10.1109/CVPR.2017.183]
Yang Y Z and Lu H. 2019. Single image deraining via recurrent hierarchy enhancement network//Proceedings of the 27th ACM International Conference on Multimedia. Nice, France: Association for Computing Machinery: 1814-1822 [DOI: 10.1145/3343031.3351149http://dx.doi.org/10.1145/3343031.3351149]
Yasarla R and Patel V M. 2019. Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 8397-8406 [DOI: 10.1109/CVPR.2019.00860http://dx.doi.org/10.1109/CVPR.2019.00860]
Yasarla R, Sindagi V A and Patel V M. 2020. Syn2real transfer learning for image deraining using gaussian processes//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 2723-2733 [DOI: 10.1109/CVPR42600.2020.00280http://dx.doi.org/10.1109/CVPR42600.2020.00280]
Yeh C H, Huang C H and Kang L W. 2020. Multi-scale deep residual learning-based single image haze removal via image decomposition. IEEE Transactions on Image Processing, 29: 3153-3167 [DOI: 10.1109/TIP.2019.2957929]
Zamir S W, Arora A, Khan S H, Hayat M, Khan F S, Yang M H and Shao L. 2021. Multi-stage progressive image restoration//Proceedings of 2021 IEEE Conference on Computer Vision and Pattern Recognition. [s. l.]: IEEE: 14821-14831
Zhang H and Patel V M. 2018. Density-aware single image de-raining using a multi-stream dense network//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 695-704 [DOI: 10.1109/CVPR.2018.00079http://dx.doi.org/10.1109/CVPR.2018.00079]
Zhang H, Sindagi V and Patel V M. 2020. Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology, 30(11): 3943-3956 [DOI: 10.1109/TCSVT.2019.2920407]
Zhang L, Zhang L, Mou X Q and Zhang D. 2011. FSIM: a feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 20(8): 2378-2386 [DOI: 10.1109/TIP.2011.2109730]
相关作者
相关机构
京公网安备11010802024621