深度特征融合注意力与双尺度的运动去模糊

陈加保; 熊邦书; 况发; 章照中

doi:10.11834/jig.220931

图像处理和编码 | 浏览量 : 0 下载量: 1 CSCD: 0

PDF
导出
分享
收藏
专辑

深度特征融合注意力与双尺度的运动去模糊
Motion deblurring based on deep feature fusion attention and double-scale
2023年28卷第12期页码：3731-3743
纸质出版日期： 2023-12-16 ，
DOI： 10.11834/jig.220931
稿件说明：

移动端阅览

陈加保，熊邦书，况发，章照中. 2023. 深度特征融合注意力与双尺度的运动去模糊. 中国图象图形学报， 28(12):3731-3743

Chen Jiabao， Xiong Bangshu， Kuang Fa， Zhang Zhaozhong. 2023. Motion deblurring based on deep feature fusion attention and double-scale. Journal of Image and Graphics， 28(12):3731-3743
陈加保，熊邦书，况发，章照中. 2023. 深度特征融合注意力与双尺度的运动去模糊. 中国图象图形学报， 28(12):3731-3743 DOI： 10.11834/jig.220931.

Chen Jiabao， Xiong Bangshu， Kuang Fa， Zhang Zhaozhong. 2023. Motion deblurring based on deep feature fusion attention and double-scale. Journal of Image and Graphics， 28(12):3731-3743 DOI： 10.11834/jig.220931.

摘要

目的

拍摄运动物体时，图像易出现运动模糊，这将影响计算机视觉任务的完成。为提升运动图像去模糊的质量，提出了基于深度特征融合注意力的双尺度去运动模糊网络。

方法

首先，设计了双尺度网络，在网络结构上设计高低尺度通路，在低尺度上增加对模糊区域的注意力，在高尺度上提升网络的高频细节恢复能力，增强了模型去模糊效果。其次，设计了深度特征融合注意力模块，通过融合全尺度特征、构建通道注意力，将编码的全尺度特征与解码的同级特征进行拼接融合，进一步增强了网络的去模糊性能和细节恢复能力。最后，在双尺度的基础上，引入多尺度损失，使模型更加关注高频细节的恢复。

结果

在3个数据集上，与12种去模糊方法进行了对比实验。在GoPro数据集上得到了最优结果，相比SRN（scale-recurrent network）方法，平均峰值信噪比提升了2.29 dB，能够恢复出更多的细节信息。在Kohler数据集上，得到了最高的峰值信噪比（29.91 dB）。在Lai数据集上，视觉上有最好的去模糊效果。

结论

实验结果表明，本文方法可以有效去除运动模糊并恢复细节。

Abstract

Objective

When taking an image， the image tends to come out blurred due to interferences from shaky or out of focus lenses， dust， or atmospheric light. Image blurs can be categorized into atmospheric， defocus， and motion blurs. The most common type of blur is motion blur， which is caused by jittering objects during shooting and has a great impact on computer vision tasks， such as image classification， object detection， text recognition. Meanwhile， methods for motion deblurring can be grouped into non-blind and blind deblurring. On the one hand， non-blind deblurring has a known fuzzy kernel and can output a sharp image due to effect of deconvolution with fuzzy image directly after removing noise. This method is simple and effective， but its premise is that the fuzzy kernel is known. On the other hand， blind deblurring faces the serious problem where the fuzzy kernel is unknown. This method needs to calculate the value of the fuzzy kernel and then deblur using the same procedure as the non-blind deblurring method. The method is complicated， computationally expensive， and time consuming. However， the fuzzy kernel is unknown in real scenarios， and most cases of deblurring are actually blind deblurring. Therefore， this paper focuses on blind deblurring. Owing to its powerful feature extraction ability， a convolutional neural network can deblur an image after obtaining its fuzzy features from a dataset of sharp and blurred image pairs， hence presenting a breakthrough for the task of image deblurring.

Method

To address motion blur， a two-scale network based on deep feature fusion attention is proposed in this paper. First， a two-scale network is designed to extract different scales of spatial information. During the transformation from high to low scale， the state of blurred features in the motion blurred image changes from smooth to sharp. Therefore， the network pays further attention to those fuzzy areas in a low-scale image， hence allowing the network to obtain fuzzy features. This network not only improves the capability of recovering frequency details from the original scale image but also effectively uses the spatial information of the blurred image to enhance the deblurring effect of the model. Second， the deep feature fusion attention module is constructed. The main structure of the network is very similar to that of U-Net. After the encoding and decoding structure， the deep feature fusion attention module is constructed to obtain the best fusion feature information. The encoding and decoding structure can obtain multi-level information about blurred images. Such information is then fed into the module of full-scale features and squeeze-and-excite in the deep feature fusion attention module to produce full-scale features， which in turn are spliced and fused with the decoded features of the same level to further enhance the recovery performance of the network. Third， in order to make the network recover the high-frequency details effectively， the function of perception loss is replaced by the function of frequency loss. The loss function in this paper is composed of two parts， namely， content loss and frequency loss. The function of content loss uses the mean absolute error and combines multi-scale knowledge to calculate the absolute disparity of each pixel between the sharp and restored images and obtain the value of content loss. This procedure also marks the first step in calculating the 2D Fourier transform of the restored and sharp images and in measuring the frequency loss. After calculating the Fourier transform， the average absolute error between the restored and sharp images is determined. The multi-scale frequency loss is obtained by multiplying the multi-scale weight by the average absolute error. Our loss function can improve the sensitivity of the network to fuzzy features and enhance the recovery ability of the model in frequency detail.

Result

We compare the performance of our model with that of 12 other methods on 3 different datasets. We evaluate the performance of our model on the GoPro dataset by utilizing the peak signal-to-noise ratio （PSNR） and structural similarity （SSIM） between the restored and sharp images. Compared with the scale-recurrent network， the proposed method obtains 2.29 dB higher PSNR， thus allowing this method to recover detailed information. We also compare the optimal results of each method. We test the generalization performance of the proposed method on the Kohler dataset， where this method achieves the highest PSNR of 29.91 dB without retraining. Meanwhile， we compare the deblurring performance of these methods for real blurred images on the Lai dataset and determine the best results via subjective comparison.

Conclusion

To improve the quality of motion deblurring， a two-scale network based on deep feature fusion attention is proposed in this paper. This work offers three contributions， namely， a novel two-scale network， a deep feature fusion attention module， and multi-scale loss. Objective and subjective experimental results show that the proposed deblurring model can efficiently integrate image spatial information and feature semantic information， thus improving its deblurring performance， PSNR， and SSIM.

关键词

深度特征融合注意力双尺度网络运动图像去模糊全尺度特征融合多尺度损失

Keywords

deep feature fusion attentiondouble-scale networkmotion image deblurringfull-scale feature fusionloss function with multi-scale

references

Ahmad J， Touqir I and Siddiqui A M. 2021. Efficient dark channel prior based blind image de-blurring. Radioengineering， 30（2）： 417-421 ［DOI： 10.13164/re.2021.0417http://dx.doi.org/10.13164/re.2021.0417］

Bahat Y， Efrat N and Irani M. 2017. Non-uniform blind deblurring by reblurring//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 3306-3314 ［DOI： 10.1109/ICCV.2017.356http://dx.doi.org/10.1109/ICCV.2017.356］

Chakrabarti A. 2016. A neural approach to blind motion deblurring//Proceedings of the 14th European Conference on Computer Vision. Amsterdam， the Netherlands： Springer： 221-235 ［DOI： 10.1007/978-3-319-46487-9_14http://dx.doi.org/10.1007/978-3-319-46487-9_14］

Cho S J， Ji S W， Hong J P， Jung S W and Ko S J. 2021. Rethinking coarse-to-fine approach in single image deblurring//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 4621-4630 ［DOI： 10.1109/ICCV48922.2021.00460http://dx.doi.org/10.1109/ICCV48922.2021.00460］

Feng H， Guo J D， Xu H X and Ge S S. 2021. SharpGAN： Dynamic scene deblurring method for smart ship based on receptive field block and generative adversarial networks. Sensors， 21（11）： #3641 ［DOI： 10.3390/s21113641http://dx.doi.org/10.3390/s21113641］

Fish E， Weinbren J and Gilbert A. 2021. Rethinking genre classification with fine grained semantic clustering//Proceedings of 2021 IEEE International Conference on Image Processing. Anchorage， USA： IEEE： 1274-1278 ［DOI： 10.1109/ICIP42928.2021.9506751http://dx.doi.org/10.1109/ICIP42928.2021.9506751］

Fuoli D， Van Gool L and Timofte R. 2021. Fourier space losses for efficient perceptual image super-resolution//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 2340-2349 ［DOI： 10.1109/ICCV48922.2021.00236http://dx.doi.org/10.1109/ICCV48922.2021.00236］

Gao H Y， Tao X， Shen X Y and Jia J Y. 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 3843-3851 ［DOI： 10.1109/CVPR.2019.00397http://dx.doi.org/10.1109/CVPR.2019.00397］

Huang H M， Lin L F， Tong R F， Hu H J， Zhang Q W， Iwamoto Y， Han X H， Chen Y W and Wu J. 2020. Unet3+： a full-scale connected unet for medical image segmentation//Proceedings of 2020 IEEE International Conference on Acoustics， Speech and Signal Processing. Barcelona， Spain： IEEE： 1055-1059 ［DOI： 10.1109/ICASSP40776.2020.9053405http://dx.doi.org/10.1109/ICASSP40776.2020.9053405］

Ju M Y， Ding C， Guo C A， Ren W Q and Tao D C. 2021. IDRLP： image dehazing using region line prior. IEEE Transactions on Image Processing， 30： 9043-9057 ［DOI： 10.1109/TIP.2021.3122088http://dx.doi.org/10.1109/TIP.2021.3122088］

Kim K， Lee S and Cho S. 2023. MSSNet： multi-scale-stage network for single image deblurring//Proceedings of 2023 European Conference on Computer Vision. Tel Aviv， Israel： Springer： 524-539 ［DOI： 10.1007/978-3-031-25063-7_32http://dx.doi.org/10.1007/978-3-031-25063-7_32］

Köhler R， Hirsch M， Mohler B， Schölkopf B and Harmeling S. 2012. Recording and playback of camera shake： benchmarking blind deconvolution with a real-world database//Proceedings of the 12th European Conference on Computer Vision. Florence， Italy： Springer： 27-40 ［DOI： 10.1007/978-3-642-33786-4_3http://dx.doi.org/10.1007/978-3-642-33786-4_3］

Kupyn O， Budzan V， Mykhailych M， Mishkin D and Matas J. 2018. DeblurGAN： blind motion deblurring using conditional adversarial networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 8183-8192 ［DOI： 10.1109/CVPR.2018.00854http://dx.doi.org/10.1109/CVPR.2018.00854］

Kupyn O， Martyniuk T， Wu J R and Wang Z Y. 2019. DeblurGAN-v2： Deblurring （orders-of-magnitude） faster and better//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision （ICCV）. Seoul， Korea （South）： IEEE： 8877-8886 ［DOI： 10.1109/ICCV.2019.00897http://dx.doi.org/10.1109/ICCV.2019.00897］

Lai W S， Huang J B， Hu Z， Ahuja N and Yang M H. 2016. A comparative study for single image blind deblurring//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Las Vegas， USA： IEEE： 1701-1709 ［DOI： 10.1109/CVPR.2016.188http://dx.doi.org/10.1109/CVPR.2016.188］

Liu J， Tan J Q and He L. 2022. Gradient cepstrum combined with simplified extreme channel prior for blind deconvolution. Circuits， Systems， and Signal Processing， 41（2）： 1074-1099 ［DOI： 10.1007/s00034-021-01827-1http://dx.doi.org/10.1007/s00034-021-01827-1］

Liu Y G， Yu J Z and Han Y H. 2018. Understanding the effective receptive field in semantic image segmentation. Multimedia Tools and Applications， 77（17）： 22159-22171 ［DOI： 10.1007/s11042-018-5704-3http://dx.doi.org/10.1007/s11042-018-5704-3］

Nah S， Kim T H and Lee M K. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 257-265 ［DOI： 10.1109/CVPR.2017.35http://dx.doi.org/10.1109/CVPR.2017.35］

Oyedotun O K， Al Ismaeil K and Aouada D. 2021. Training very deep neural networks： rethinking the role of skip connections. Neurocomputing， 441： 105-117 ［DOI： 10.1016/j.neucom.2021.02.004http://dx.doi.org/10.1016/j.neucom.2021.02.004］

Pan J S， Sun D Q， Pfister H and Yang M H. 2016. Blind image deblurring using dark channel prior//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 1628-1636 ［DOI： 10.1109/CVPR.2016.180http://dx.doi.org/10.1109/CVPR.2016.180］

Park D， Kang D U， Kim J and Chun S Y. 2020. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 327-343 ［DOI： 10.1007/978-3-030-58539-6_20http://dx.doi.org/10.1007/978-3-030-58539-6_20］

Purohit K and Rajagopalan A N. 2020. Region-adaptive dense network for efficient motion deblurring//Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York， USA： AAAI： 11882-11889 ［DOI： 10.1609/aaai.v34i07.6862http://dx.doi.org/10.1609/aaai.v34i07.6862］

Qiu F， Hou F， Yuan Y and Wang W C. 2019. Blind image deblurring with reinforced use of edges. Journal of Image and Graphics， 24（6）： 847-858

邱枫，侯飞，袁野，王文成. 2019. 加强边缘感知的盲去模糊算法. 中国图象图形学报， 24（6）： 847-858 ［DOI： 10.11834/jig.180543http://dx.doi.org/10.11834/jig.180543］

Šroubek F and Kotera J. 2020. Motion blur prior//Proceedings of 2020 IEEE International Conference on Image Processing （ICIP）. Abu Dhabi， United Arab Emirates： IEEE： 928-932 ［DOI： 10.1109/ICIP40778.2020.9191316http://dx.doi.org/10.1109/ICIP40778.2020.9191316］

Suin M， Purohit K and Rajagopalan A N. 2020. Spatially-attentive patch-hierarchical network for adaptive motion deblurring//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 3603-3612 ［DOI： 10.1109/CVPR42600.2020.00366http://dx.doi.org/10.1109/CVPR42600.2020.00366］

Tao X， Gao H Y， Shen X Y， Wang J and Jia J Y. 2018. Scale-recurrent network for deep image deblurring//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 8174-8182 ［DOI： 10.1109/CVPR.2018.00853http://dx.doi.org/10.1109/CVPR.2018.00853］

Tsai F J， Peng Y T， Lin Y Y， Tsai C C and Lin C W. 2021. BANet： Blur-aware attention networks for dynamic scene deblurring ［EB/OL］. ［2022-06-22］. https://arxiv.org/pdf/2101.07518.pdfhttps://arxiv.org/pdf/2101.07518.pdf

Wang W S and Su C. 2022. An optimization method for motion blur image restoration and ringing suppression via texture mapping. ISA Transactions， 131： 650-661 ［DOI： 10.1016/j.isatra.2022.05.005http://dx.doi.org/10.1016/j.isatra.2022.05.005］

Wu D， Zhao H T and Zheng S B. 2020. Motion deblurring method based on DenseNets. Journal of Image and Graphics， 25（5）： 890-899

吴迪，赵洪田，郑世宝. 2020. 密集连接卷积网络图像去模糊. 中国图象图形学报， 25（5）： 890-899 ［DOI： 10.11834/jig.190400http://dx.doi.org/10.11834/jig.190400］

Yang L G and Ji H. 2019. A variational EM framework with adaptive edge selection for blind motion deblurring//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 10159-10168 ［DOI： 10.1109/CVPR.2019.01041http://dx.doi.org/10.1109/CVPR.2019.01041］

Ye M Y， Lyu D and Chen G S. 2020. Scale-iterative upscaling network for image deblurring. IEEE Access， 8： 18316-18325 ［DOI： 10.1109/ACCESS.2020.2967823http://dx.doi.org/10.1109/ACCESS.2020.2967823］

Zamir S W， Arora A， Khan S， Hayat M， Khan F S， Yang M H and Shao L. 2021. Multi-stage progressive image restoration//Proceeding of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 14816-14826 ［DOI： 10.1109/CVPR46437.2021.01458http://dx.doi.org/10.1109/CVPR46437.2021.01458］

Zhang H G， Dai Y C， Li H D and Koniusz P. 2019. Deep stacked hierarchical multi-patch network for image deblurring//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 5971-5979 ［DOI： 10.1109/CVPR.2019.00613http://dx.doi.org/10.1109/CVPR.2019.00613］

Zhang J W， Pan J S， Ren J， Song Y B， Bao L C， Lau R W H and Yang M H. 2018. Dynamic scene deblurring using spatially variant recurrent neural networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 2521-2529 ［DOI： 10.1109/CVPR.2018.00267http://dx.doi.org/10.1109/CVPR.2018.00267］

Zhang K H， Luo W H， Zhong Y R， Ma L， Stenger B， Liu W and Li H D. 2020. Deblurring by realistic blurring//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 2734-2743 ［DOI： 10.1109/CVPR42600.2020.00281http://dx.doi.org/10.1109/CVPR42600.2020.00281］

文章被引用时，请邮件提醒。

提交

暂无数据