浏览全部资源
扫码关注微信
收稿：2019-12-30，
修回：2020-3-30，
录用：2020-4-6，
纸质出版：2020-12-16
移动端阅览
目的
2
雨天户外采集的图像常常因为雨线覆盖图像信息产生色变和模糊现象。为了提高雨天图像的质量，本文提出一种基于自适应选择卷积网络深度学习的单幅图像去雨算法。
方法
2
针对雨图中背景误判和雨痕残留问题，加入网络训练的雨线修正系数（refine factor，RF），改进现有雨图模型，更精确地描述雨图中各像素受到雨线的影响。构建选择卷积网络（selective kernel network，SK Net），自适应地选择不同卷积核对应维度的信息，进一步学习、融合不同卷积核的信息，提高网络的表达力，最后构建包含SK Net、refine factor net和residual net子网络的自适应卷积残差修正网络（selective kernel convolution using a residual refine factor，SKRF），直接学习雨线图和残差修正系数（RF），减少映射区间，减少背景误判。
结果
2
实验通过设计的SKRF网络，在公开的Rain12测试集上进行去雨实验，取得了比现有方法更高的精确度，峰值信噪比（peak signal to noise ratio，PSNR）达到34.62 dB，结构相似性（structural similarity，SSIM）达到0.970 6。表明SKRF网络对单幅图像去雨效果有明显优势。
结论
2
单幅图像去雨SKRF算法为雨图模型中的雨线图提供一个额外的修正残差系数，以降低学习映射区间，自适应选择卷积网络模型提升雨图模型的表达力和兼容性。
Objective
2
Rain lines adversely affect the visual quality of images collected from outdoors. Severe weather conditions
such as rain
fog
and haze
can affect the quality of these images and make them unusable. These degraded images may also drastically affect the performance of man's vision system. Given that rain is a common meteorological phenomenon
an algorithm that can remove rain from single image is of practical significance. Given that video-based de-raining methods obtain pixel information of the same location at different periods
removing rain from an individual image is more challenging because of less available information. Traditional de-raining methods mainly focus on rain map modeling and use mathematical optimization to detect and remove rain streaks
but the performance of such approach requires further improvement.
Method
2
To address the above problems
this paper establishes a convolution neural network for single image rain removal that is trained on a synthetic dataset. The contributions of this work are as follows. 1) To expand the neural receptive field of a convolution neural network that learns abstract feature representation of rain streaks and the ground truth
this work establishes a selective kernel network based on multi-scale convolution with different kernel for feature learning. To accomplish useful information fusion and selection
an external non-linear weight learning mechanism is developed to redistribute the weight for the corresponding channel's feature information from different convolution kernels. This mechanism enables the network to select the feature information of different receptive fields adaptively and enhance its expression ability and rain removal capability. 2) The existing rain map model shows some limitations at the training stage. Completing this model by adding a learnable refine factor that modifies each pixel in a rain streak image
can enhance the accuracy of the result and prevent background misjudgment. The range of the refining factor is also limited to reduce the mapping range of the network training process. 3) At the training stage the existing single image rain removal networks need to learn various types of image content
including rain streaks removal and background restoration
which will undoubtedly increase their burden. By using the novel idea of residual learning the proposed network can directly learn the rain streak map by using the input rain map. In this way
the mapping interval of the network learning process is reduced
the background of the original graph can be preserved
and loss of details can be prevented. The validity of the above arguments is tested by designing a comparison network with different modules. Specifically
based on general convolution
different modules are combined step by step
including the SK net
residual learning mechanism
and refine factor learning net. Single image rain removal network based on selective kernel convolution using residual refine factor (SKRF) is eventually designed. The residual learning mechanism is used to reduce the mapping interval
and the refined factor is used to enhance the rain streak map to improve the rain removal performance.
Result
2
An SKRF network
including the three subnets of SK net
refine factor net
and residual net
is designed in a rain removal experiment and tested on the open Rain12 test set. This network achieves a higher accuracy
peak signal to noise ratio(PSNR) (34.62)
and structural similarity(SSIM) (0.970 6) compared with the existing methods. The SKRF network shows obvious advantages in removing rain from single image.
Conclusion
2
We construct a convolution neural network based on SKRF to remove rain streaks from single image. A selective kernel convolution network is established to improve the expression ability of the proposed network via the adaptive adjustment mechanism of the size of the receptive field by the internal neurons. A rain map with different characteristics can be well learned
and the effect of rain removal can be improved. The residual learning mechanism can reduce the mapping interval of the network learning process and retain more details of the original image. In the modified rain map model
an additional refine factor is provided for the rain streak map
which can further reduce the mapping interval and reduce background misjudgment. This network not only removes the majority of the visible rain streaks but also retains the ground truth. In our feature work
we plan to extend this network to a wider range of image restoration tasks.
Dong C, Loy C C, He K M and Tang X O. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295-307[DOI:10.1109/TPAMI.2015.2439281]
Fu X Y, Huang J B, Ding X H, Liao Y H and Paisley J. 2017a. Clearing the skies:a deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6):2944-2956[DOI:10.1109/TIP.2017.2691802]
Fu X Y, Huang J B, Zeng D L, Huang Y and Paisley J. 2017b. Removing raoh from single images via a deep detail network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Honolulu, USA: IEEE: 3855-3863
Fu X Y, Liang B R, Huang Y, Ding X H and Paisley J. 2019. Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems, 31(6):1794-1807[DOI:10.1109/TNNLS.2019.2926481]
Goodfellow I J, Pouget-Abadie J and Mirza M. 2014. Generative adversarial networks//Advances in Neural Information Processing Systems, 3: 2672-2680[ DOI: 10.13140/RG.2.2.31946.62401 http://dx.doi.org/10.13140/RG.2.2.31946.62401 ]
He K M, Zhang X Y, Ren S Q and Sun J. 2016a. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
He K M, Zhang X Y, Ren S Q and Sun J. 2016b. Identity mappings in deep residual networks//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 630-645[ DOI: 10.1007/978-3-319-46493-0_38 http://dx.doi.org/10.1007/978-3-319-46493-0_38 ]
Howard A G, Zhu M L, Chen B, Kalenichenko D, Wang W J, Weyand T, Andreetto M and Adam H. 2017. Mobilenets: efficient convolutional neural networks for mobile vision applications[EB/OL]. https://arxiv.org/pdf/1704.04861.pdf https://arxiv.org/pdf/1704.04861.pdf
Hu J, Shen L, Albanie S, Sun G and Wu E.2018. Squeeze-and-excitation networks//Proceedings of 2018 IEEE Conference on Computer vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7132-7141[ DOI: 10.1109/CVPR.2018.00745 http://dx.doi.org/10.1109/CVPR.2018.00745 ]
Huang G, Liu Z, van der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2261-2269[ DOI: 10.1109/CVPR.2017.243 http://dx.doi.org/10.1109/CVPR.2017.243 ]
Huynh Thu Q and Ghanbari M. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics Letters, 44(13):800-801[DOI:10.1049/el:20080522]
Ioffe S and Szegedy C. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift//Proceedings of the 32nd International Conference on International Conference on Machine Learning. Lille, France: ACM: 448-456
Kang L W, Lin C W and Fu Y H. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, 21(4):1742-1755[DOI:10.1109/TIP.2011.2179057]
Kang L W, Lin C W and Fu Y H.2011.Automatic single-image-based rain streaks removal via image decomposition.//IEEE Transactions on Image Processing, 2011, 21(4): 1742-1755[ DOI: 10.1109/TIP.2011.2179057 http://dx.doi.org/10.1109/TIP.2011.2179057 ]
Li X, Wang W H, Hu X L and Yang J. 2019. Selective kernel networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 510-519[ DOI: 10.1109/CVPR.2019.00060 http://dx.doi.org/10.1109/CVPR.2019.00060 ]
Li Y, Tan R T, Guo X J, Li J B and Brown M S. 2016. Rain streak removal using layer priors//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 2736-2744[ DOI: 10.1109/CVPR.2016.299 http://dx.doi.org/10.1109/CVPR.2016.299 ]
Lin T Y, Dollár P, Girshick R, He K M, Hariharan B and Belongie S. 2017. Feature pyramid networks for object detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2117-2125[ DOI: 10.1109/CVPR.2017.106 http://dx.doi.org/10.1109/CVPR.2017.106 ]
Luo Y, Xu Y and Ji H. 2015. Removing rain from a single image via discriminative sparse coding//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 3397-3405[ DOI: 10.1109/ICCV.2015.388 http://dx.doi.org/10.1109/ICCV.2015.388 ]
Nair V and Hinton G E. 2010. Rectified linear units improve restricted Boltzmann machines//Proceedings of the 27th International Conference on International Conference on Machine Learning. Haifa, Israel: Omnipress: 807-814
Ren S Q, He K M, Girshick R and Sun J. 2017a. Faster R-CNN:towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-1149[DOI:10.1109/TPAMI.2016.2577031]
Ren W H, Tian J D, Han Z, Chan A and Tang Y D. 2017b. Video desnowing and deraining based on matrix decomposition//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2838-3847[ DOI: 10.1109/CVPR.2017.303 http://dx.doi.org/10.1109/CVPR.2017.303 ]
Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition[EB/OL]. https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf
Szegedy C, Vanhoucke V, Ioffe S, Shlens J and Wojna Z. 2016. Rethinking the inception architecture for computer vision//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 2818-2826[ DOI: 10.1109/CVPR.2016.308 http://dx.doi.org/10.1109/CVPR.2016.308 ]
Wang M H, Mai J M, Cai R C, Liang Y and Wan H. 2018. Single image deraining using deep convolutional networks. Multimedia Tools and Applications, 77(19):25905-25918[DOI:10.1007/s11042-018-5825-8]
Wang M, Chen L, Liang Y, Hao Y, He H and Li C. 2020. Single image rain removal with reusing original input squeeze-and-excitation network. IET Image Processing: 1467-1474[ DOI: 10.1049/iet-ipr.2019.0716 http://dx.doi.org/10.1049/iet-ipr.2019.0716 ]
Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment:from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612[DOI:10.1109/TIP.2003.819861]
Yang W H, Tan R T, Feng J S, Liu J Y, Guo Z M and Yan S C. 2017. Deep joint rain detection and removal from a single image//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 1685-1694[ DOI: 10.1109/CVPR.2017.183 http://dx.doi.org/10.1109/CVPR.2017.183 ]
Yu F, Koltun V and Funkhouser T. 2017. Dilated residual networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 636-644[ DOI: 10.1109/CVPR.2017.75 http://dx.doi.org/10.1109/CVPR.2017.75 ]
Zhang H and Patel V M. 2018. Density-aware single image de-raining using a multi-stream dense network//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE:1685-1694[ DOI: 10.1109/CVPR.2018.00079 http://dx.doi.org/10.1109/CVPR.2018.00079 ]
Zhang H, Sindagi V and Patel V M. 2019. Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology: #99[ DOI: 10.1109/TCSVT.2019.2920407 http://dx.doi.org/10.1109/TCSVT.2019.2920407 ]
相关作者
相关机构
京公网安备11010802024621
