浏览全部资源
扫码关注微信
收稿：2023-02-23，
修回：2023-07-17，
纸质出版：2024-01-16
移动端阅览
目的
2
现有低照度图像增强方法大多依赖于像素级重建，旨在学习低照度输入和正常曝光图像之间的确定性映射，没有对复杂的光照分布进行建模，从而导致了不适当的亮度及噪声。大多图像生成方法仅使用一种（显式或隐式）生成模型，在灵活性和效率方面有所限制。为此，改进了一种混合显式—隐式的生成模型，该模型允许同时进行对抗训练和最大似然训练。
方法
2
首先设计了一个残差注意力条件编码器对低照度输入进行处理，提取丰富的特征以减少生成图像的色差；然后，将编码器提取到的特征作为可逆流生成模型的条件先验，学习将正常曝光图像的分布映射为高斯分布的双向映射，以此来模拟正常曝光图像的条件分布，使模型能够对多个正常曝光结果进行采样，生成多样化的样本；最后，利用隐式生成对抗网络（generative adversarial network，GAN）为模型提供约束，改善图像的细节信息。特别地，两个映射方向都受到损失函数的约束，因此本文设计的模型具有较强的抗模式崩溃能力。
结果
2
实验在2个数据集上进行训练与测试，在低照度（low-light dataset，LOL）数据集与其他算法对比，本文算法在峰值信噪比（peak signal-to-noise ratio，PSNR）上均有最优表现、图像感知相似度（learned perceptual image patch similarity，LPIPS）、在结构相似性（structural similarity index measure，SSIM）上取得次优表现0.01，在无参考自然图像质量指标（natural image quality evaluator，NIQE）上取得较优结果。具体地，相较于18种现有显著性模型中的最优值，本文算法PSNR提高0.84 dB，LPIPS降低0.02，SSIM降低0.01，NIQE值降低1.05。在 MIT-Adobe FiveK（Massachu⁃setts Institute of Technology Adobe FiveK）数据集中，与5种显著性模型进行对比，相较于其中的最优值，本文算法PSNR提高0.58 dB，SSIM值取得并列第一。
结论
2
本文提出的流生成对抗模型，综合了显式和隐式生成模型的优点，更好地调整了低照度图像的光照，抑制了噪声和伪影，提高了生成图像的视觉感知质量。
Objective
2
Low-light images are produced by imaging devices that cannot capture sufficient light due to unavoidable environmental or technical limitations （such as nighttime， backlight， and underexposure）. Such images usually have the characteristics of low brightness， low contrast， narrow grayscale range， color distortion， and strong noise， which almost need more information. Low-light images containing these problems do not meet human beings’ visual requirements and directly limit the role of the subsequent advanced visual system. The low-light image enhancement task is an ill-posed problem because the low-light image loss of illumination information， that is， a low-light image may correspond to countless normal-light images. Low-light image enhancement should be regarded as selecting the most suitable solution from all possible outputs. Most existing reconstruction methods rely on pixel-level reconstruction algorithms that aim to learn a deterministic mapping between low-light inputs and normal-light images. They provide a normal-light result for a low-light image rather than modeling complex lighting distributions， which usually result in inappropriate brightness and noise. Furthermore， most existing image generation methods use only one （explicit or implicit） generative model， which limits flexibility and efficiency. Flow models have recently demonstrated promising results for low-level vision tasks. This paper improves a hybrid explicit-implicit generative model， which can flexibly and efficiently reconstruct normal-light images with satisfied lighting， cleanliness， and realism from degraded inputs. The model alleviates the fuzzy details and singularity problems produced by explicit or implicit generative modeling.
Method
2
This paper proposes a low-light image enhancement network with a hybrid explicit （Flow） and implicit generative adversarial network （GAN）， named LLFlowGAN that contains three parts： conditional encoder， flow generation network， and discriminator. Flow generation networks operate at multiple scales conditioned on encoded information from low-light input. First， a residual attention conditional encoder is designed to process low-light input， calculate low-light color maps， and extract rich features to reduce the color deviation of generated images. Due to the flexibility of the flow model， the conditional encoder mainly consists of several residual blocks plus efficient stacking of channel attention modules. Then， the features extracted by the encoder are used as conditional prior to the generative flow model. Moreover， the flow model learns to map the high-dimensional random variables obeying the normal exposure image distribution into a bidirectional mapping with simple tractable latent variables （Gaussian distribution）. By simulating the conditional distribution of normal exposure images， the model allows the sampling of multiple normal exposure results to generate diverse samples. Finally， the GAN-based discriminator provides constraints for the model and improves the detailed information of the image in the reverse mapping. Because the model learns a bidirectional mapping relationship， both mapping directions can be regarded as constrained by the loss function， providing the network stability and resistance to mode collapse.
Result
2
The proposed algorithm in this paper is validated using experiments on two datasets， namely， Low-Light （LOL） dataset and MIT-Adobe FiveK dataset， to verify its effectiveness. Quantitative evaluation metrics include peak signal-to-noise ratio （PSNR）， structural similarity index measure （SSIM）， learned perceptual image patch similarity （LPIPS）， and natural image quality evaluator （NIQE）. Our model is compared with 18 saliency models in the LOL dataset， including the traditional supervised and unsupervised deep learning methods including state-of-the-art methods in this field. Compared with the model with the second-best performance， our method improves the PSNR value by 0.84 dB and reduces the LPIPS value （the smaller， the better） by 0.02. SSIM obtains the second-best value， decreases by 0.01， and NIQE decreases by 1.05. Saliency maps of each method are also provided for comparison. Our method better preserves rich detail and color information while enhancing image brightness， where artifacts are rarely observed， achieving better perceptual quality. In the MIT-Adobe FiveK dataset， the five most advanced methods are compared. Compared with the model with the second-best performance， the PSNR value increases by 0.58 dB， and the SSIM value is also tied for first place. In addition， a series of ablation experiments and cross-dataset tests in the LOL dataset are conducted to verify the effectiveness of each algorithm module. Experimental results prove our proposed algorithm improves the effect of low-light image enhancement.
Conclusion
2
In this paper， a hybrid explicit-implicit generative model is proposed. The model inherits the flow-based explicit generative model， which can accurately complete the free conversion between the natural image space and a simple Gaussian distribution and flexibly generate diverse samples. The adversarial training strategy is further used to improve the detailed information of the generated image， enrich the saturation， and reduce the color distortion. The proposed approach can achieve competitive performance compared with representative state-of-the-art low-light image enhancement methods.
Ardizzone L ， Lüth C ， Kruse J ， Rother C and Köthe U . 2019 . Guided image generation with conditional invertible neural networks ［EB/OL］. ［ 2023-02-01 ］. https://arxiv.org/pdf/1907.02392.pdf https://arxiv.org/pdf/1907.02392.pdf
Bychkovsky V ， Paris S ， Chan E and Durand F . 2011 . Learning photographic global tonal adjustment with a database of input/output image pairs // Proceedings of 2011 CVPR . Colorado Springs， USA ： IEEE： 97 - 104 ［ DOI： 10.1109/CVPR.2011.5995413 http://dx.doi.org/10.1109/CVPR.2011.5995413 ］
Chen Y S ， Wang Y C ， Kao M H and Chuang Y Y . 2018 . Deep photo enhancer： unpaired learning for image enhancement from photographs with GANs // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 6306 - 6314 ［ DOI： 10.1109/CVPR.2018.00660 http://dx.doi.org/10.1109/CVPR.2018.00660 ］
Dai L Q and Tang J H . 2022 . iFlowGAN： an invertible flow-based generative adversarial network for unsupervised image-to-image translation . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 44 （ 8 ）： 4151 - 4162 ［ DOI： 10.1109/TPAMI.2021.3062849 http://dx.doi.org/10.1109/TPAMI.2021.3062849 ］
Fan C M ， Liu T J and Liu K H . 2022 . Half wavelet attention on M-Net+ for low-light image enhancement // Proceedings of 2022 IEEE International Conference on Image Processing . Bordeaux， France ： IEEE： 3878 - 3882 ［ DOI： 10.1109/ICIP46576.2022.9897503 http://dx.doi.org/10.1109/ICIP46576.2022.9897503 ］
Fu X Y ， Zeng D L ， Huang Y ， Liao Y H ， Ding X H and Paisley J . 2016 . A fusion-based enhancing method for weakly illuminated images . Signal Processing ， 129 ： 82 - 96 ［ DOI： 10.1016/j.sigpro.2016.05.031 http://dx.doi.org/10.1016/j.sigpro.2016.05.031 ］
Gao Y Y ， Hu H M ， Li B and Guo Q . 2018 . Naturalness preserved nonuniform illumination estimation for image enhancement based on retinex . IEEE Transactions on Multimedia ， 20 （ 2 ）： 335 - 344 ［ DOI： 10.1109/tmm.2017.2740025 http://dx.doi.org/10.1109/tmm.2017.2740025 ］
Grover A ， Dhar M and Ermon S . 2018 . Flow-GAN： combining maximum likelihood and adversarial learning in generative models . Proceedings of the AAAI Conference on Artificial Intelligence ， 32 （ 1 ）： 3069 - 3076 ［ DOI： 10.1609/aaai.v32i1.11829 http://dx.doi.org/10.1609/aaai.v32i1.11829 ］
Guo C L ， Li C Y ， Guo J C ， Loy C C ， Hou J H ， Kwong S and Cong R M . 2020 . Zero-reference deep curve estimation for low-light image enhancement // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 1777 - 1786 ［ DOI： 10.1109/CVPR42600.2020.00185 http://dx.doi.org/10.1109/CVPR42600.2020.00185 ］
Guo X J and Hu Q M . 2023 . Low-light image enhancement via breaking down the darkness . International Journal of Computer Vision ， 131 （ 2 ）： 48 - 66 ［ DOI： 10.1007/s11263-022-01667-9 http://dx.doi.org/10.1007/s11263-022-01667-9 ］
Guo X J ， Li Y and Ling H B . 2017 . LIME： low-light image enhancement via illumination map estimation . IEEE Transactions on Image Processing ， 26 （ 2 ）： 982 - 993 ［ DOI： 10.1109/tip.2016.2639450 http://dx.doi.org/10.1109/tip.2016.2639450 ］
Jiang Y F ， Gong X Y ， Liu D ， Cheng Y ， Fang C ， Shen X H ， Yang J C ， Zhou P and Wang Z Y . 2021 . EnlightenGAN： deep light enhancement without paired supervision . IEEE Transactions on Image Processing ， 30 ： 2340 - 2349 ［ DOI： 10.1109/tip.2021.3051462 http://dx.doi.org/10.1109/tip.2021.3051462 ］
Jobson D J ， Rahman Z and Woodell G A . 1997a . A multiscale retinex for bridging the gap between color images and the human observation of scenes . IEEE Transactions on Image Processing ， 6 （ 7 ）： 965 - 976 ［ DOI： 10.1109/83.597272 http://dx.doi.org/10.1109/83.597272 ］
Jobson D J ， Rahman Z and Woodell G A . 1997b . Properties and performance of a center/surround retinex . IEEE Transactions on Image Processing ， 6 （ 3 ）： 451 - 462 ［ DOI： 10.1109/83.557356 http://dx.doi.org/10.1109/83.557356 ］
Johnson J ， Alahi A and Li F F . 2016 . Perceptual losses for real-time style transfer and super-resolution // Proceedings of the 14th European Conference on Computer Vision . Amsterdam， the Netherlands ： Springer： 694 - 711 ［ DOI： 10.1007/978-3-319-46475-6_43 http://dx.doi.org/10.1007/978-3-319-46475-6_43 ］
Kingma D P and Dhariwal P . 2018 . Glow： generative flow with invertible 1 x 1 convolutions ［EB/OL］. ［ 2023-02-01 ］. https://arxiv.org/pdf/1807.03039.pdf https://arxiv.org/pdf/1807.03039.pdf
Ledig C ， Theis L ， Huszr F ， Caballero J ， Cunningham A ， Acosta A ， Aitken A ， Tejani A ， Totz J ， Wang Z H and Shi W Z . 2017 . Photo-Realistic single image super-resolution using a generative adversarial network // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 105 - 114 ［ DOI： 10.1109/cvpr.2017.19 http://dx.doi.org/10.1109/cvpr.2017.19 ］
Liang J X ， Xu Y ， Quan Y H ， Shi B X and Ji H . 2022 . Self-supervised low-light image enhancement using discrepant untrained network priors . IEEE Transactions on Circuits and Systems for Video Technology ， 32 （ 11 ）： 7332 - 7345 ［ DOI： 10.1109/tcsvt.2022.3181781 http://dx.doi.org/10.1109/tcsvt.2022.3181781 ］
Liu R S ， Ma L ， Zhang J A ， Fan X and Luo Z X . 2021a . Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville， USA ： IEEE： 10556 - 10565 ［ DOI： 10.1109/cvpr46437.2021.01042 http://dx.doi.org/10.1109/cvpr46437.2021.01042 ］
Liu Y ， Qin Z Y ， Anwar S ， Ji P ， Kim D ， Caldwell S and Gedeon T . 2021b . Invertible denoising network： a light solution for real noise removal // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville， USA ： IEEE： 13360 - 13369 ［ DOI： 10.1109/cvpr46437.2021.01316 http://dx.doi.org/10.1109/cvpr46437.2021.01316 ］
Lore K G ， Akintayo A and Sarkar S . 2017 . LLNet： a deep autoencoder approach to natural low-light image enhancement . Pattern Recognition ， 61 ： 650 - 662 ［ DOI： /10.1016/j.patcog.2016.06.008 http://dx.doi.org//10.1016/j.patcog.2016.06.008 ］
Lugmayr A ， Danelljan M ， Van Gool L and Timofte R . 2020 . SRFlow： learning the super-resolution space with normalizing flow // Proceedings of the 16th Computer Vision . Glasgow， UK ： Springer： 715 - 732 ［ DOI： 10.1007/978-3-030-58558-7_42 http://dx.doi.org/10.1007/978-3-030-58558-7_42 ］
Ma L ， Ma T Y and Liu R S . 2022 . The review of low-light image enhancement . Journal of Image and Graphics ， 27 （ 5 ）： 1392 - 1409
马龙 ， 马腾宇 ， 刘日升 . 2022 . 低光照图像增强算法综述 . 中国图象图形学报 ， 27 （ 5 ）： 1392 - 1409 ［ DOI： 10.11834/jig.210852 http://dx.doi.org/10.11834/jig.210852 ］
Wang J ， Chen K J ， Zhang W M and Yu N H . 2023 . Image processing network-inverted identifiable secure natural steganography . Journal of Image and Graphics ， 28 （ 3 ）： 749 - 759
王健 ， 陈可江 ， 张卫明 ， 俞能海 . 2023 . 面向可逆图像处理网络的可证安全自然隐写 . 中国图象图形学报 ， 28 （ 3 ）： 749 - 759 ［ DOI： 10.11834/jig.220529 http://dx.doi.org/10.11834/jig.220529 ］
Wang R X ， Zhang Q ， Fu C W ， Shen X Y ， Zheng W S and Jia J Y . 2019a . Underexposed photo enhancement using deep illumination estimation // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 6842 - 6850 ［ DOI： 10.1109/cvpr.2019.00701 http://dx.doi.org/10.1109/cvpr.2019.00701 ］
Wang X T ， Yu K ， Wu S X ， Gu J J ， Liu Y H ， Dong C ， Qiao Y and Loy C C . 2019b . ESRGAN： enhanced super-resolution generative adversarial networks // Proceedings of the Computer Vision —— ECCV 2018 Workshops . Munich， Germany ： Springer： 63 - 79 ［ DOI： 10.1007/978-3-030-11021-5_5 http://dx.doi.org/10.1007/978-3-030-11021-5_5 ］
Wang Y F ， Wan R J ， Yang W H ， Li H L ， Chau L P and Kot A . 2022 . Low-light image enhancement with normalizing flow . Proceedings of the AAAI Conference on Artificial Intelligence ， 36 （ 3 ）： 2604 - 2612 ［ DOI： 10.1609/aaai.v36i3.20162 http://dx.doi.org/10.1609/aaai.v36i3.20162 ］
Wei C ， Wang W J ， Yang W H and Liu J Y . 2018 . Deep retinex decomposition for low-light enhancement ［EB/OL］. ［ 2023-02-01 ］. https://arxiv.org/pdf/1808.04560.pdf https://arxiv.org/pdf/1808.04560.pdf
Xiao M Q ， Zheng S X ， Liu C ， Wang Y L ， He D ， Ke G L ， Bian J ， Lin Z C and Liu T Y . 2020 . Invertible image rescaling // Proceedings of the 16th European Conference on Computer Vision —— ECCV 2020 . Glasgow， UK ： Springer： 126 - 144 ［ DOI： 10.1007/978-3-030-58452-8_8 http://dx.doi.org/10.1007/978-3-030-58452-8_8 ］
Yang W H ， Wang S Q ， Fang Y M ， Wang Y and Liu J Y . 2020 . From fidelity to perceptual quality： a semi-supervised approach for low-light image enhancement // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 3060 - 3069 ［ DOI： 10.1109/cvpr42600.2020.00313 http://dx.doi.org/10.1109/cvpr42600.2020.00313 ］
Ying Z Q ， Li G ， Ren Y R ， Wang R G and Wang W M . 2017 . A new low-light image enhancement algorithm using camera response model // Proceedings of 2017 IEEE International Conference on Computer Vision Workshops . Venice， Italy ： IEEE： 3015 - 3022 ［ DOI： 10.1109/iccvw.2017.356 http://dx.doi.org/10.1109/iccvw.2017.356 ］
Zamir S W ， Arora A ， Khan S ， Hayat M ， Khan F S ， Yang M H and Shao L . 2020 . Learning enriched features for real image restoration and enhancement // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 492 - 511 ［ DOI： 10.1007/978-3-030-58595-2_30 http://dx.doi.org/10.1007/978-3-030-58595-2_30 ］
Zhang Y H ， Guo X J ， Ma J Y ， Liu W and Zhang J W . 2021 . Beyond brightening low-light images . International Journal of Computer Vision ， 129 （ 4 ）： 1013 - 1037 ［ DOI： 10.1007/s11263-020-01407-x http://dx.doi.org/10.1007/s11263-020-01407-x ］
Zhang Y H ， Zhang J W and Guo X J . 2019 . Kindling the darkness： a practical low-light image enhancer // Proceedings of the 27th ACM International Conference on Multimedia . Nice， France ： ACM： 1632 - 1640 ［ DOI： 10.1145/3343031.3350926 http://dx.doi.org/10.1145/3343031.3350926 ］
Zhao Z J ， Xiong B S ， Wang L ， Ou Q F ， Yu L and Kuang F . 2022 . RetinexDIP： a unified deep framework for low-light image enhancement . IEEE Transactions on Circuits and Systems for Video Technology ， 32 （ 3 ）： 1076 - 1088 ［ DOI： 10.1109/tcsvt.2021.3073371 http://dx.doi.org/10.1109/tcsvt.2021.3073371 ］
Zhou Z G ， Sang N and Hu X R . 2014 . Global brightness and local contrast adaptive enhancement for low illumination color image . Optik ， 125 （ 6 ）： 1795 - 1799 ［ DOI： 10.1016/j.ijleo.2013.09.051 http://dx.doi.org/10.1016/j.ijleo.2013.09.051 ］
Zhu Y R ， Huang J ， Fu X Y ， Zhao F ， Sun Q B and Zha Z J . 2022 . Bijective mapping network for shadow removal // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 5617 - 5626 ［ DOI： 10.1109/cvpr52688.2022.00554 http://dx.doi.org/10.1109/cvpr52688.2022.00554 ］
Zuiderveld K . 1994 . Contrast limited Adaptive histogram equalization // Graphics Gems . San Diego， USA ： Academic Press： 474 - 485 ［ DOI： 10.1016/B978-0-12-336156-1.50061-6 http://dx.doi.org/10.1016/B978-0-12-336156-1.50061-6 ］
相关作者
相关机构
京公网安备11010802024621
