浏览全部资源
扫码关注微信
纸质出版日期： 2024-01-16 ，
移动端阅览
黄颖， 彭慧， 李昌盛， 高胜美， 陈奉. 2024. LLFlowGAN：以生成对抗方式约束可逆流的低照度图像增强. 中国图象图形学报， 29(01):0065-0079
Huang Ying， Peng Hui， Li Changsheng， Gao Shengmei， Chen Feng. 2024. LLFlowGAN： a low-light image enhancement method for constraining invertible flow in a generative adversarial manner. Journal of Image and Graphics， 29(01):0065-0079
目的
2
现有低照度图像增强方法大多依赖于像素级重建，旨在学习低照度输入和正常曝光图像之间的确定性映射，没有对复杂的光照分布进行建模，从而导致了不适当的亮度及噪声。大多图像生成方法仅使用一种（显式或隐式）生成模型，在灵活性和效率方面有所限制。为此，改进了一种混合显式—隐式的生成模型，该模型允许同时进行对抗训练和最大似然训练。
方法
2
首先设计了一个残差注意力条件编码器对低照度输入进行处理，提取丰富的特征以减少生成图像的色差；然后，将编码器提取到的特征作为可逆流生成模型的条件先验，学习将正常曝光图像的分布映射为高斯分布的双向映射，以此来模拟正常曝光图像的条件分布，使模型能够对多个正常曝光结果进行采样，生成多样化的样本；最后，利用隐式生成对抗网络（generative adversarial network，GAN）为模型提供约束，改善图像的细节信息。特别地，两个映射方向都受到损失函数的约束，因此本文设计的模型具有较强的抗模式崩溃能力。
结果
2
实验在2个数据集上进行训练与测试，在低照度（low-light dataset，LOL）数据集与其他算法对比，本文算法在峰值信噪比（peak signal-to-noise ratio，PSNR）上均有最优表现、图像感知相似度（learned perceptual image patch similarity，LPIPS）、在结构相似性（structural similarity index measure，SSIM）上取得次优表现0.01，在无参考自然图像质量指标（natural image quality evaluator，NIQE）上取得较优结果。具体地，相较于18种现有显著性模型中的最优值，本文算法PSNR提高0.84 dB，LPIPS降低0.02，SSIM降低0.01，NIQE值降低1.05。在 MIT-Adobe FiveK（Massachu⁃setts Institute of Technology Adobe FiveK）数据集中，与5种显著性模型进行对比，相较于其中的最优值，本文算法PSNR提高0.58 dB，SSIM值取得并列第一。
结论
2
本文提出的流生成对抗模型，综合了显式和隐式生成模型的优点，更好地调整了低照度图像的光照，抑制了噪声和伪影，提高了生成图像的视觉感知质量。
Objective
2
Low-light images are produced by imaging devices that cannot capture sufficient light due to unavoidable environmental or technical limitations （such as nighttime， backlight， and underexposure）. Such images usually have the characteristics of low brightness， low contrast， narrow grayscale range， color distortion， and strong noise， which almost need more information. Low-light images containing these problems do not meet human beings’ visual requirements and directly limit the role of the subsequent advanced visual system. The low-light image enhancement task is an ill-posed problem because the low-light image loss of illumination information， that is， a low-light image may correspond to countless normal-light images. Low-light image enhancement should be regarded as selecting the most suitable solution from all possible outputs. Most existing reconstruction methods rely on pixel-level reconstruction algorithms that aim to learn a deterministic mapping between low-light inputs and normal-light images. They provide a normal-light result for a low-light image rather than modeling complex lighting distributions， which usually result in inappropriate brightness and noise. Furthermore， most existing image generation methods use only one （explicit or implicit） generative model， which limits flexibility and efficiency. Flow models have recently demonstrated promising results for low-level vision tasks. This paper improves a hybrid explicit-implicit generative model， which can flexibly and efficiently reconstruct normal-light images with satisfied lighting， cleanliness， and realism from degraded inputs. The model alleviates the fuzzy details and singularity problems produced by explicit or implicit generative modeling.
Method
2
This paper proposes a low-light image enhancement network with a hybrid explicit （Flow） and implicit generative adversarial network （GAN）， named LLFlowGAN that contains three parts： conditional encoder， flow generation network， and discriminator. Flow generation networks operate at multiple scales conditioned on encoded information from low-light input. First， a residual attention conditional encoder is designed to process low-light input， calculate low-light color maps， and extract rich features to reduce the color deviation of generated images. Due to the flexibility of the flow model， the conditional encoder mainly consists of several residual blocks plus efficient stacking of channel attention modules. Then， the features extracted by the encoder are used as conditional prior to the generative flow model. Moreover， the flow model learns to map the high-dimensional random variables obeying the normal exposure image distribution into a bidirectional mapping with simple tractable latent variables （Gaussian distribution）. By simulating the conditional distribution of normal exposure images， the model allows the sampling of multiple normal exposure results to generate diverse samples. Finally， the GAN-based discriminator provides constraints for the model and improves the detailed information of the image in the reverse mapping. Because the model learns a bidirectional mapping relationship， both mapping directions can be regarded as constrained by the loss function， providing the network stability and resistance to mode collapse.
Result
2
The proposed algorithm in this paper is validated using experiments on two datasets， namely， Low-Light （LOL） dataset and MIT-Adobe FiveK dataset， to verify its effectiveness. Quantitative evaluation metrics include peak signal-to-noise ratio （PSNR）， structural similarity index measure （SSIM）， learned perceptual image patch similarity （LPIPS）， and natural image quality evaluator （NIQE）. Our model is compared with 18 saliency models in the LOL dataset， including the traditional supervised and unsupervised deep learning methods including state-of-the-art methods in this field. Compared with the model with the second-best performance， our method improves the PSNR value by 0.84 dB and reduces the LPIPS value （the smaller， the better） by 0.02. SSIM obtains the second-best value， decreases by 0.01， and NIQE decreases by 1.05. Saliency maps of each method are also provided for comparison. Our method better preserves rich detail and color information while enhancing image brightness， where artifacts are rarely observed， achieving better perceptual quality. In the MIT-Adobe FiveK dataset， the five most advanced methods are compared. Compared with the model with the second-best performance， the PSNR value increases by 0.58 dB， and the SSIM value is also tied for first place. In addition， a series of ablation experiments and cross-dataset tests in the LOL dataset are conducted to verify the effectiveness of each algorithm module. Experimental results prove our proposed algorithm improves the effect of low-light image enhancement.
Conclusion
2
In this paper， a hybrid explicit-implicit generative model is proposed. The model inherits the flow-based explicit generative model， which can accurately complete the free conversion between the natural image space and a simple Gaussian distribution and flexibly generate diverse samples. The adversarial training strategy is further used to improve the detailed information of the generated image， enrich the saturation， and reduce the color distortion. The proposed approach can achieve competitive performance compared with representative state-of-the-art low-light image enhancement methods.
低照度图像增强流模型生成对抗网络（GAN）双向映射复杂光照分布
low-light images enhancementflow modelgenerative adversarial network （GAN）bidirectional mappingcomplex illumination distribution
Ardizzone L， Lüth C， Kruse J， Rother C and Köthe U. 2019. Guided image generation with conditional invertible neural networks ［EB/OL］. ［2023-02-01］. https://arxiv.org/pdf/1907.02392.pdfhttps://arxiv.org/pdf/1907.02392.pdf
Bychkovsky V， Paris S， Chan E and Durand F. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs//Proceedings of 2011 CVPR. Colorado Springs， USA： IEEE： 97-104 ［DOI： 10.1109/CVPR.2011.5995413http://dx.doi.org/10.1109/CVPR.2011.5995413］
Chen Y S， Wang Y C， Kao M H and Chuang Y Y. 2018. Deep photo enhancer： unpaired learning for image enhancement from photographs with GANs//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 6306-6314 ［DOI： 10.1109/CVPR.2018.00660http://dx.doi.org/10.1109/CVPR.2018.00660］
Dai L Q and Tang J H. 2022. iFlowGAN： an invertible flow-based generative adversarial network for unsupervised image-to-image translation. IEEE Transactions on Pattern Analysis and Machine Intelligence， 44（8）： 4151-4162 ［DOI： 10.1109/TPAMI.2021.3062849http://dx.doi.org/10.1109/TPAMI.2021.3062849］
Fan C M， Liu T J and Liu K H. 2022. Half wavelet attention on M-Net+ for low-light image enhancement//Proceedings of 2022 IEEE International Conference on Image Processing. Bordeaux， France： IEEE： 3878-3882 ［DOI： 10.1109/ICIP46576.2022.9897503http://dx.doi.org/10.1109/ICIP46576.2022.9897503］
Fu X Y， Zeng D L， Huang Y， Liao Y H， Ding X H and Paisley J. 2016. A fusion-based enhancing method for weakly illuminated images. Signal Processing， 129： 82-96 ［DOI： 10.1016/j.sigpro.2016.05.031http://dx.doi.org/10.1016/j.sigpro.2016.05.031］
Gao Y Y， Hu H M， Li B and Guo Q. 2018. Naturalness preserved nonuniform illumination estimation for image enhancement based on retinex. IEEE Transactions on Multimedia， 20（2）： 335-344 ［DOI： 10.1109/tmm.2017.2740025http://dx.doi.org/10.1109/tmm.2017.2740025］
Grover A， Dhar M and Ermon S. 2018. Flow-GAN： combining maximum likelihood and adversarial learning in generative models. Proceedings of the AAAI Conference on Artificial Intelligence， 32（1）： 3069-3076 ［DOI： 10.1609/aaai.v32i1.11829http://dx.doi.org/10.1609/aaai.v32i1.11829］
Guo C L， Li C Y， Guo J C， Loy C C， Hou J H， Kwong S and Cong R M. 2020. Zero-reference deep curve estimation for low-light image enhancement//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 1777-1786 ［DOI： 10.1109/CVPR42600.2020.00185http://dx.doi.org/10.1109/CVPR42600.2020.00185］
Guo X J and Hu Q M. 2023. Low-light image enhancement via breaking down the darkness. International Journal of Computer Vision， 131（2）： 48-66 ［DOI： 10.1007/s11263-022-01667-9http://dx.doi.org/10.1007/s11263-022-01667-9］
Guo X J， Li Y and Ling H B. 2017. LIME： low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing， 26（2）： 982-993 ［DOI： 10.1109/tip.2016.2639450http://dx.doi.org/10.1109/tip.2016.2639450］
Jiang Y F， Gong X Y， Liu D， Cheng Y， Fang C， Shen X H， Yang J C， Zhou P and Wang Z Y. 2021. EnlightenGAN： deep light enhancement without paired supervision. IEEE Transactions on Image Processing， 30： 2340-2349 ［DOI： 10.1109/tip.2021.3051462http://dx.doi.org/10.1109/tip.2021.3051462］
Jobson D J， Rahman Z and Woodell G A. 1997a. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing， 6（7）： 965-976 ［DOI： 10.1109/83.597272http://dx.doi.org/10.1109/83.597272］
Jobson D J， Rahman Z and Woodell G A. 1997b. Properties and performance of a center/surround retinex. IEEE Transactions on Image Processing， 6（3）： 451-462 ［DOI： 10.1109/83.557356http://dx.doi.org/10.1109/83.557356］
Johnson J， Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam， the Netherlands： Springer： 694-711 ［DOI： 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43］
Kingma D P and Dhariwal P. 2018. Glow： generative flow with invertible 1x1 convolutions ［EB/OL］. ［2023-02-01］. https://arxiv.org/pdf/1807.03039.pdfhttps://arxiv.org/pdf/1807.03039.pdf
Ledig C， Theis L， Huszr F， Caballero J， Cunningham A， Acosta A， Aitken A， Tejani A， Totz J， Wang Z H and Shi W Z. 2017. Photo-Realistic single image super-resolution using a generative adversarial network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 105-114 ［DOI： 10.1109/cvpr.2017.19http://dx.doi.org/10.1109/cvpr.2017.19］
Liang J X， Xu Y， Quan Y H， Shi B X and Ji H. 2022. Self-supervised low-light image enhancement using discrepant untrained network priors. IEEE Transactions on Circuits and Systems for Video Technology， 32（11）： 7332-7345 ［DOI： 10.1109/tcsvt.2022.3181781http://dx.doi.org/10.1109/tcsvt.2022.3181781］
Liu R S， Ma L， Zhang J A， Fan X and Luo Z X. 2021a. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 10556-10565 ［DOI： 10.1109/cvpr46437.2021.01042http://dx.doi.org/10.1109/cvpr46437.2021.01042］
Liu Y， Qin Z Y， Anwar S， Ji P， Kim D， Caldwell S and Gedeon T. 2021b. Invertible denoising network： a light solution for real noise removal//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 13360-13369 ［DOI： 10.1109/cvpr46437.2021.01316http://dx.doi.org/10.1109/cvpr46437.2021.01316］
Lore K G， Akintayo A and Sarkar S. 2017. LLNet： a deep autoencoder approach to natural low-light image enhancement. Pattern Recognition， 61： 650-662 ［DOI：/10.1016/j.patcog.2016.06.008http://dx.doi.org//10.1016/j.patcog.2016.06.008］
Lugmayr A， Danelljan M， Van Gool L and Timofte R. 2020. SRFlow： learning the super-resolution space with normalizing flow//Proceedings of the 16th Computer Vision. Glasgow， UK： Springer： 715-732 ［DOI： 10.1007/978-3-030-58558-7_42http://dx.doi.org/10.1007/978-3-030-58558-7_42］
Ma L， Ma T Y and Liu R S. 2022. The review of low-light image enhancement. Journal of Image and Graphics， 27（5）： 1392-1409
马龙， 马腾宇， 刘日升. 2022. 低光照图像增强算法综述. 中国图象图形学报， 27（5）： 1392-1409 ［DOI： 10.11834/jig.210852http://dx.doi.org/10.11834/jig.210852］
Wang J， Chen K J， Zhang W M and Yu N H. 2023. Image processing network-inverted identifiable secure natural steganography. Journal of Image and Graphics， 28（3）： 749-759
王健， 陈可江， 张卫明， 俞能海. 2023. 面向可逆图像处理网络的可证安全自然隐写. 中国图象图形学报， 28（3）： 749-759 ［DOI： 10.11834/jig.220529http://dx.doi.org/10.11834/jig.220529］
Wang R X， Zhang Q， Fu C W， Shen X Y， Zheng W S and Jia J Y. 2019a. Underexposed photo enhancement using deep illumination estimation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 6842-6850 ［DOI： 10.1109/cvpr.2019.00701http://dx.doi.org/10.1109/cvpr.2019.00701］
Wang X T， Yu K， Wu S X， Gu J J， Liu Y H， Dong C， Qiao Y and Loy C C. 2019b. ESRGAN： enhanced super-resolution generative adversarial networks//Proceedings of the Computer Vision —— ECCV 2018 Workshops. Munich， Germany： Springer： 63-79 ［DOI： 10.1007/978-3-030-11021-5_5http://dx.doi.org/10.1007/978-3-030-11021-5_5］
Wang Y F， Wan R J， Yang W H， Li H L， Chau L P and Kot A. 2022. Low-light image enhancement with normalizing flow. Proceedings of the AAAI Conference on Artificial Intelligence， 36（3）： 2604-2612 ［DOI： 10.1609/aaai.v36i3.20162http://dx.doi.org/10.1609/aaai.v36i3.20162］
Wei C， Wang W J， Yang W H and Liu J Y. 2018. Deep retinex decomposition for low-light enhancement ［EB/OL］. ［2023-02-01］. https://arxiv.org/pdf/1808.04560.pdfhttps://arxiv.org/pdf/1808.04560.pdf
Xiao M Q， Zheng S X， Liu C， Wang Y L， He D， Ke G L， Bian J， Lin Z C and Liu T Y. 2020. Invertible image rescaling//Proceedings of the 16th European Conference on Computer Vision —— ECCV 2020. Glasgow， UK： Springer： 126-144 ［DOI： 10.1007/978-3-030-58452-8_8http://dx.doi.org/10.1007/978-3-030-58452-8_8］
Yang W H， Wang S Q， Fang Y M， Wang Y and Liu J Y. 2020. From fidelity to perceptual quality： a semi-supervised approach for low-light image enhancement//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 3060-3069 ［DOI： 10.1109/cvpr42600.2020.00313http://dx.doi.org/10.1109/cvpr42600.2020.00313］
Ying Z Q， Li G， Ren Y R， Wang R G and Wang W M. 2017. A new low-light image enhancement algorithm using camera response model//Proceedings of 2017 IEEE International Conference on Computer Vision Workshops. Venice， Italy： IEEE： 3015-3022 ［DOI： 10.1109/iccvw.2017.356http://dx.doi.org/10.1109/iccvw.2017.356］
Zamir S W， Arora A， Khan S， Hayat M， Khan F S， Yang M H and Shao L. 2020. Learning enriched features for real image restoration and enhancement//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 492-511 ［DOI： 10.1007/978-3-030-58595-2_30http://dx.doi.org/10.1007/978-3-030-58595-2_30］
Zhang Y H， Guo X J， Ma J Y， Liu W and Zhang J W. 2021. Beyond brightening low-light images. International Journal of Computer Vision， 129（4）： 1013-1037 ［DOI： 10.1007/s11263-020-01407-xhttp://dx.doi.org/10.1007/s11263-020-01407-x］
Zhang Y H， Zhang J W and Guo X J. 2019. Kindling the darkness： a practical low-light image enhancer//Proceedings of the 27th ACM International Conference on Multimedia. Nice， France： ACM： 1632-1640 ［DOI： 10.1145/3343031.3350926http://dx.doi.org/10.1145/3343031.3350926］
Zhao Z J， Xiong B S， Wang L， Ou Q F， Yu L and Kuang F. 2022. RetinexDIP： a unified deep framework for low-light image enhancement. IEEE Transactions on Circuits and Systems for Video Technology， 32（3）： 1076-1088 ［DOI： 10.1109/tcsvt.2021.3073371http://dx.doi.org/10.1109/tcsvt.2021.3073371］
Zhou Z G， Sang N and Hu X R. 2014. Global brightness and local contrast adaptive enhancement for low illumination color image. Optik， 125（6）： 1795-1799 ［DOI： 10.1016/j.ijleo.2013.09.051http://dx.doi.org/10.1016/j.ijleo.2013.09.051］
Zhu Y R， Huang J， Fu X Y， Zhao F， Sun Q B and Zha Z J. 2022. Bijective mapping network for shadow removal//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans， USA： IEEE： 5617-5626 ［DOI： 10.1109/cvpr52688.2022.00554http://dx.doi.org/10.1109/cvpr52688.2022.00554］
Zuiderveld K. 1994. Contrast limited Adaptive histogram equalization//Graphics Gems. San Diego， USA： Academic Press： 474-485 ［DOI： 10.1016/B978-0-12-336156-1.50061-6http://dx.doi.org/10.1016/B978-0-12-336156-1.50061-6］
相关作者
相关机构
京公网安备11010802024621