融合注意力机制和上下文信息的微光图像增强

赵兴运; 孙帮勇

doi:10.11834/jig.210583

低照图像增强 | 浏览量 : 0 下载量: 1080 CSCD: 0

PDF
导出
分享
收藏
专辑

融合注意力机制和上下文信息的微光图像增强
An attention mechanism and contextual information based low-light image enhancement method
2022年27卷第5期页码：1565-1576
收稿：2021-07-19，

修回：2021-9-15，

录用：2021-9-22，

纸质出版：2022-05-16
DOI： 10.11834/jig.210583
稿件说明：

移动端阅览

赵兴运, 孙帮勇. 融合注意力机制和上下文信息的微光图像增强[J]. 中国图象图形学报, 2022,27(5):1565-1576. DOI： 10.11834/jig.210583.

Xingyun Zhao, Bangyong Sun. An attention mechanism and contextual information based low-light image enhancement method[J]. Journal of Image and Graphics, 2022, 27(5): 1565-1576. DOI： 10.11834/jig.210583.

摘要

目的

微光图像存在低对比度、噪声伪影和颜色失真等退化问题，造成图像的视觉感受质量较差，同时也导致后续图像识别、分类和检测等任务的精度降低。针对以上问题，提出一种融合注意力机制和上下文信息的微光图像增强方法。

方法

为提高运算精度，以U型结构网络为基础构建了一种端到端的微光图像增强网络框架，主要由注意力机制编/解码模块、跨尺度上下文模块和融合模块等组成。由混合注意力块(包括空间注意力和通道注意力)引导主干网络学习，其空间注意力模块用于计算空间位置的权重以学习不同区域的噪声特征，而通道注意力模块根据不同通道的颜色信息计算通道权重，以提升网络的颜色信息重建能力。此外，跨尺度上下文模块用于聚合各阶段网络中的深层和浅层特征，借助融合机制来提高网络的亮度和颜色增强效果。

结果

本文方法与现有主流方法进行定量和定性对比实验，结果显示本文方法显著提升了微光图像亮度，并且较好保持了图像颜色一致性，原微光图像较暗区域的噪点显著去除，重建图像的纹理细节清晰。在峰值信噪比(peak signal-to-noise ratio，PSNR)、结构相似性(structural similarity，SSIM)和图像感知相似度(perceptual image patch similarity，LPIPS)等客观指标上，本文方法较其他方法的最优值分别提高了0.74 dB、0.153和0.172。

结论

本文方法能有效解决微光图像存在的曝光不足、噪声干扰和颜色不一致等问题，具有一定应用价值。

Abstract

Objective

Low-light images capturing is common used in the night or dark scenario. In a low-light imaging scenario

the number of photons penetrating the lens is tiny

resulting in poor image quality

poor visibility

low contrast

more noise

and color distortion. Low-light images constrain many subsequent image processing tasks on the aspects of image classification

target recognition

intelligent monitoring and object recognition. The current low-light level image enhancement technology has mainly two challenging issues as following: 1) space-independent image brightness; 2) non-uniform noise. The spatial feature distribution of real low-light images is complex

and the number of penetrating photons varies greatly at different spatial positions

resulting in strong light-related variability in the image space. Existing deep learning methods can effectively improve the illumination characteristics in artificially generated datasets

but the overall image visibility and the enhancement effect of underexposed areas need to be improved for the space-restricted low illumination. For instance

de-noising will ignore some image details before image enhancement. It is difficult to reconstruct high-noise pixel information

and de-noising will easily lead to blurred images after enhancement. Our attention mechanism and contextual information based low-light image enhancement method can fully enhance the image to suppress potential noise and maintain color consistency.

Method

Our demonstration constructs an end-to-end low-light enhancement network

using U-Net as the basic framework

which mainly includes channel attention modules

encoders

decoders

cross-scale context modules and feature fusion modules. The backbone network is mainly guided by mixed attention block. The input is the low-light image to be enhanced

and the output is an enhanced high-quality noise-free color image of the same size. After the low-light enhancement network extracts the shallow features of the input original low-light image

it first uses the channel attention module to learn the weight of each channel

and assigns more weight to the useful color information channel to extract more effective image color features. Then

the channel weight and the original low-light image are used as the input of the encoder

and the semantic features of the image are extracted by the mixed attention block. Mixed attention block extracts input features from spatial information and channel information respectively. It is beneficial to restore the brightness and color information of the image and noise suppression via the location of noise in the image and the color features of different channels. In the decoder part

the de-convolution module restores the semantic features extracted by mixed attention block to high-resolution images. In addition

a cross-scale context module is designed to fuse cross-scale skip connections based on the skip connections performed on the corresponding scale features of conventional encoders and decoders.

Result

In respect of qualitative evaluation

our method is slightly higher than the reference image in fitting the overall contrast of the image

but it is better than all other contrast methods in terms of color details and noise suppression. In terms of quantitative evaluation indicators of image quality

including peak signal-to-noise ratio (PSNR)

structural similarity (SSIM) and perceptual image patch similarity (LPIPS). The methods proposed in this paper have also achieved excellent results

which are 0.74 dB

0.153 and 0.172 higher than the optimal values of other methods. The results show that our proposed method can enhance the brightness of the image and maintain the consistency of the image color

better remove the influence of noise in the dark area of the image

and naturally retain the texture details of the image.

Conclusion

Our method has its priority in brightness enhancement

noise suppression

image texture structure and color consistency.

关键词

Keywords

references

Cao H, Wang Y Y, Chen J, Jiang D S, Zhang X P, Tian Q and Wang M N. 2021. Swin-Unet: unet-like pure transformer for medical image segmentation[EB/OL]. [2021-06-27] . https://arxiv.org/pdf/2105.05537.pdf https://arxiv.org/pdf/2105.05537.pdf

Celik T. 2014. Spatial entropy-based global and local image contrast enhancement. IEEE Transactions on Image Processing, 23(12): 5298-5308 [DOI: 10.1109/TIP.2014.2364537]

Chen C, Chen Q F, Xu J and Koltun V. 2018. Learning to see in the dark//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 3291-3300 [ DOI: 10.1109/CVPR.2018.00347 http://dx.doi.org/10.1109/CVPR.2018.00347 ]

Guo C L, Li C Y, Guo J C, Loy C C, Hou J H, Kwong S and Cong R M. 2020. Zero-reference deep curve estimation for low-light image enhancement//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 1777-1786 [ DOI: 10.1109/CVPR42600.2020.00185 http://dx.doi.org/10.1109/CVPR42600.2020.00185 ]

Guo X J, Li Y and Ling H B. 2017. LIME: low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 26(2): 982-993 [DOI: 10.1109/TIP.2016.2639450]

Huang H, Tao H J and Wang H F. 2019. Low-illumination image enhancement using a conditional generative adversarial network. Journal of Image and Graphics, 24(12): 2149-2158

黄鐄, 陶海军, 王海峰. 2019. 条件生成对抗网络的低照度图像增强方法. 中国图象图形学报, 24(12): 2149-2158 [DOI: 10.11834/jig.190145]

Jiang Y F, Gong X Y, Liu D, Cheng Y, Fang C, Shen X H, Yang J C, Zhou P and Wang Z Y. 2021. EnlightenGAN: deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30: 2340-2349 [DOI: 10.1109/TIP.2021.3051462]

Jobson D J, Rahman Z and Woodell G A. 1997a. Properties and performance of a center/surround retinex. IEEE Transactions on Image Processing, 6(3): 451-462 [DOI: 10.1109/83.557356]

Jobson D J, Rahman Z and Woodell G A. 1997b. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing, 6(7): 965-976 [DOI: 10.1109/83.597272]

Johnson J, Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 694-711 [ DOI: 10.1007/978-3-319-46475-6_43 http://dx.doi.org/10.1007/978-3-319-46475-6_43 ]

Kim W, Jeong J and You J. 2012. Contrast enhancement using histogram equalization based on logarithmic mapping. Optical Engineering, 51(6): #067002 [DOI: 10.1117/1.OE.51.6.067002]

Lai W S, Huang J B, Ahuja N and Yang M H. 2019. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(11): 2599-2613 [DOI: 10.1109/TPAMI.2018.2865304]

Lee C, Lee C and Kim C S. 2013. Contrast enhancement based on layered difference representation of 2D histograms. IEEE Transactions on Image Processing, 22(12): 5372-5384 [DOI: 10.1109/TIP.2013.2284059]

Li Q H, Bi D Y, Ma S P and He Y B. 2010. Image enhancement based on Retinex and vision adaptability. Journal of Image and Graphics, 15(12): 1728-1732

李权合, 毕笃彦, 马时平, 何宜宝. 2010. 基于Retinex和视觉适应性的图像增强. 中国图象图形学报, 15(12): 1728-1732 [DOI: 10.11834/jig.20101221]

Li M D, Liu J Y, Yang W H, Sun X Y and Guo Z M. 2018. Structure-revealing low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing, 27(6): 2828-2841 [DOI: 10.1109/TIP.2018.2810539]

Lore K G, Akintayo A and Sarkar S. 2017. LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, 61: 650-662 [DOI: 10.1016/j.patcog.2016.06.008]

Lyu F F, Lu F, Wu J H and Lim C. 2018. MBLLEN: low-light image/video enhancement using CNNs//Proceedings of the 29th British Machine Vision Conference 2018. Newcastle, UK: BMVA Press: 1-13

Ma K D, Zeng K and Wang Z. 2015. Perceptual quality assessment for multi-exposure image fusion. IEEE Transactions on Image Processing, 24(11): 3345-3356 [DOI: 10.1109/TIP.2015.2442920]

Vonikakis V, Andreadis I and Gasteratos A. 2008. Fast centre-surround contrast modification. IET Image Processing, 2(1): 19-34 [DOI: 10.1049/IET-IPR:20070012]

Wang R X, Zhang Q, Fu C W, Shen X Y, Zheng W S and Jia J Y. 2019. Underexposed photo enhancement using deep illumination estimation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 6842-6850 [ DOI: 10.1109/CVPR.2019.00701 http://dx.doi.org/10.1109/CVPR.2019.00701 ]

Wang S H, Zheng J, Hu H M and Li B. 2013. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, 22(9): 3538-3548 [DOI: 10.1109/TIP.2013.2261309]

Wang W J, Wei C, Yang W H and Liu J Y. 2018. GLADNet: low-light enhancement network with global awareness//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi′an, China: IEEE: 751-755 [ DOI: 10.1109/FG.2018.00118 http://dx.doi.org/10.1109/FG.2018.00118 ]

Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612 [DOI: 10.1109/TIP.2003.819861]

Wei C, Wang W J, Yang W H and Liu J Y. 2018. Deep retinex decomposition for low-light enhancement//Proceedings of the 29th British Machine Vision Conference 2018. Newcastle, UK: BMVA Press: 1-12

Zhang R, Isola P, Efros A A, Shechtman E and Wang O. 2018. The unreasonable effectiveness of deep features as a perceptual metric//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 586-595 [ DOI: 10.1109/CVPR.2018.00068 http://dx.doi.org/10.1109/CVPR.2018.00068 ]

Zhang Y, Di X G, Zhang B and Wang C H. 2020. Self-supervised image enhancement network: training with low light images only[EB/OL]. [2021-06-27] . https://arxiv.org/pdf/2002.11300.pdf https://arxiv.org/pdf/2002.11300.pdf

Zhang Y H, Zhang J W and Guo X J. 2019. Kindling the darkness: a practical low-light image enhancer//Proceedings of the 27th ACM International Conference on Multimedia. Nice, France: ACM Press: 1632-1640 [ DOI: 10.1145/3343031.3350926 http://dx.doi.org/10.1145/3343031.3350926 ]