Rain streak removal using learnable hybrid MAP network

Long Ma; Risheng Liu; Zhiying Jiang; Yiyang Wang; Xin Fan; Haojie Li

doi:10.11834/jig.170390

Column of China MM 2017 | Views : 0 下载量: 7 CSCD: 3

PDF
Export
Share
Collection
Album

Rain streak removal using learnable hybrid MAP network
Vol. 23, Issue 2, Pages: 277-285(2018)
Received：29 July 2017，

Revised：2017-9-27，

Published：16 February 2018
DOI： 10.11834/jig.170390
稿件说明：

移动端阅览

DOI：

Long Ma, Risheng Liu, Zhiying Jiang, Yiyang Wang, Xin Fan, Haojie Li. Rain streak removal using learnable hybrid MAP network[J]. Journal of Image and Graphics, 2018, 23(2): 277-285. DOI： 10.11834/jig.170390.

摘要

目的

近年来，人工智能成为新兴研究领域，得到越来越多研究机构的关注。图像恢复问题一直是低层次计算机视觉领域的一个研究热点，其中，图像去雨由于其雨线分布的未知性及其求解的病态性，导致难以解决。现有方法存在雨线和背景之间的估计具有依赖性，难以平衡雨线去除效果与估计背景的清晰程度之间的关系；局限性比较大，训练数据很难涵盖各种场景下的雨图，而测试结果受训练数据的影响，导致难于泛化。针对上述不足，借鉴一般图像恢复问题思路，将模型与以数据驱动的网络相结合，凸显网络与模型各自的优势，提出可学习的混合MAP网络有效地解决图像去雨问题。

方法

首先基于最大后验估计（MAP）建立含有隐式先验的能量模型，然后通过优化算法将模型分解为背景估计模型和雨线估计模型两部分，以减少背景估计和雨线估计之间的依赖性。对于背景估计模型，通过对模型及优化目标分析采用以数据驱动的去噪残差网络进行建模，保证估计出的背景更清晰；对于雨线估计模型，为避免直接对未知的雨线建模失去准确性，利用高斯混合模型实时刻画输入雨图的雨线先验。

结果

在合成数据集Rain12及真实雨图上进行实验，通过综合考虑定量分析和定性分析，并与3种基于模型的方法及两种基于深度网络的方法相比，本文方法在去除雨线的同时能够损失的背景信息最少，合成数据集上的平均结构相似性（SSIM）值达到0.92。

结论

本文通过将基于模型的方法与基于深度网络的方法相结合，既去除了雨线又保证了估计背景的清晰程度，同时也验证了将传统模型与深度网络相结合是一种解决图像恢复问题的有效途径。

Abstract

Objective

In recent years

artificial intelligence has become the emerging area and attract the attention of many research institutions. Computer vision is the force at the core of the development in the area of artificial intelligence. Image restoration remains an active research topic in low-level computer vision. Because the distribution of the rain streaks is irregular and the problem is ill-posed

rain removal for the single image is one of the difficult problems to be solved. So far

the basic idea to solve the problem is building the rational rain image generative model

and find the solution by different measures. There are two representative ways. One is by describing the priors of rain streaks and background with conventional models and then solve it by the optimization algorithm

but the shortcoming is due to the interdependent connection between rain streaks and the background

it is hard to balance the relationship between the performance of rain removal and the clarity of background. The other is deep neural network based framework to build the network structure and loss function

then learning the network with large amounts of data

but the shortcoming is the difficulty of generalization because of test results are subject to the training data which is hard to cover the rain in the different scene.

Method

For the shortcomings mentioned above

inspired by general image restoration

this paper combines the network by data-driven with the empirical model

and the process displays the advantages of the network and empirical model

then learnable mixture MAP network is proposed so that image deraining problem can be solved effectively. The specific process is to build the energy model which contains the implicit prior item based on maximum a posterior (MAP) framework

then transfer the model to two parts that is estimating background and estimating rain streaks by the optimization algorithm

and that reduce the dependency of background and rain streaks. For the first part

it can be solved by the residual network for denoising by analysis the model and the optimization objective and in this way

the estimated image is more clearly. For the second part

the rain streaks prior to the input image can be described real-time by gaussian mixture model to improve the accuracy of estimating rain streaks.

Result

In the end

compared with the three empirical model-based algorithms and two deep learning-based algorithms

the experiments on the synthetic dataset and real-world image demonstrate the good performance of our model on removing rain streaks meanwhile reducing the information loss of the background

in addition

the SSIM value reaches to 0.92 on the synthetic dataset.

Conclusion

This paper proposes to combine model-based method with deep network-based method

assimilates the advantages from each other. Not only can the majority of visible rain streaks be removed

but also remains detail information of background. Achieving a harmonious of the two aspects and standing out from state-of-the-art methods. At the same time

this paper makes a verification

which is the effective way to solve the image restoration problem by combining the conventional model and deep network.

关键词

Keywords

references

Kang L W, Lin C W, Fu Y H. Automatic single-image-based rain streaks removal via image decomposition[J]. IEEE Transactions on Image Processing, 2012, 21(4):1742-1755.[DOI:10.1109/TIP.2011.2179057]

Chen Y L, Hsu C T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks[C]//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, NSW, Australia: IEEE, 2013: 1968-1975. [ DOI:10.1109/ICCV.2013.247 http://dx.doi.org/10.1109/ICCV.2013.247 ]

Kim J H, Lee C, Sim J Y, et al. Single-image deraining using an adaptive nonlocal means filter[C]//Proceedings of the 20th IEEE International Conference on Image Processing. Melbourne, VIC, Australia: IEEE, 2013: 914-917. [ DOI:10.1109/ICIP.2013.6738189 http://dx.doi.org/10.1109/ICIP.2013.6738189 ]

Luo Y, Xu Y, Ji H. Removing rain from a single image via discriminative sparse coding[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 3397-3405. [ DOI:10.1109/ICCV.2015.388 http://dx.doi.org/10.1109/ICCV.2015.388 ]

Li Y, Tan R T, Guo X J, et al. Rain streak removal using layer priors[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 2736-2744. [ DOI:10.1109/CVPR.2016.299 http://dx.doi.org/10.1109/CVPR.2016.299 ]

Fu X Y, Huang J B, Ding X H, et al. Clearing the skies:a deep network architecture for single-image rain removal[J]. IEEE Transactions on Image Processing, 2017, 26(6):2944-2956.[DOI:10.1109/TIP.2017.2691802]

Yang W H, Tan R T, Feng J S, et al. Deep joint rain detection and removal from a single image[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 1357-1366. [ DOI:10.1109/CVPR.2017.183 http://dx.doi.org/10.1109/CVPR.2017.183 ]

Fu X Y, Huang J B, Zeng D L, et al. Removing rain from single images via a deep detail network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 3855-3863. [ DOI:10.1109/CVPR.2017.186 http://dx.doi.org/10.1109/CVPR.2017.186 ]

Wang Y L, Yang J F, Yin W T, et al. A new alternating minimization algorithm for total variation image reconstruction[J]. SIAM Journal on Imaging Sciences, 2008, 1(3):248-272.[DOI:10.1137/080724265]

Geman D, Yang C D. Nonlinear image recovery with half-quadratic regularization[J]. IEEE Transactions on Image Processing, 1995, 4(7):932-946.[DOI:10.1109/83.392335]

Glowinski R, Le Tallec P. Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics[M]. Philadelphia, Pennsylvania:Society for Industrial and Applied Mathematics, 1989.

Boyd S, Parikh N, Chu E, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trends ® in Machine Learning, 2011, 3(1):1-122.[DOI:10.1561/2200000016]

Chan S H, Wang X R, Elgendy O A. Plug-and-play ADMM for image restoration:Fixed-point convergence and applications[J]. IEEE Transactions on Computational Imaging, 2017, 3(1):84-98.[DOI:10.1109/TCI.2016.2629286]

Chen Y J, Pock T. Trainable nonlinear reaction diffusion:a flexible framework for fast and effective image restoration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1256-1272.[DOI:10.1109/TPAMI.2016.2596743]

Zoran D, Weiss Y. From learning models of natural image patches to whole image restoration[C]//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona, Spain: IEEE, 2011: 479-486. [ DOI:10.1109/ICCV.2011.6126278 http://dx.doi.org/10.1109/ICCV.2011.6126278 ]

Garg K, Nayar S K. Photorealistic rendering of rain streaks[J]. ACM Transactions on Graphics, 2006, 25(3):996-1002.[DOI:10.1145/1141911.1141985]

Martin D, Fowlkes C, Malik J, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of 8th IEEE International Conference on Computer Vision. Vancouver, BC, Canada: IEEE, 2001: 416-423. [ DOI:10.1109/ICCV.2001.937655 http://dx.doi.org/10.1109/ICCV.2001.937655 ]

Alert me when the article has been cited

提交

Image classification network with random dilated convolution

Double-pooling residual classification network based on feature reordering attention mechanism

Chemical structure recognition method based on attention mechanism and encoder-decoder architecture

RTDNet： red tide detection network for high-resolution satellite images

A micro-hyperspectral image classification method of gallbladder cancer based on multi-scale fusion attention mechanism