Vegetation and sky probability mask generation for visible and near-infrared image fusion

Can Tong; Jiacheng Ying; Huiliang Shen

doi:10.11834/jig.210805

Image Analysis and Recognition | Views : 0 下载量: 70 CSCD: 0

PDF
Export
Share
Collection
Album

Vegetation and sky probability mask generation for visible and near-infrared image fusion
Vol. 27, Issue 12, Pages: 3516-3529(2022)
Received：24 September 2021，

Revised：2021-11-26，

Accepted：01 December 2021，

Published：16 December 2022
DOI： 10.11834/jig.210805
稿件说明：

移动端阅览

DOI：

Can Tong, Jiacheng Ying, Huiliang Shen. Vegetation and sky probability mask generation for visible and near-infrared image fusion[J]. Journal of Image and Graphics, 2022, 27(12): 3516-3529. DOI： 10.11834/jig.210805.

摘要

目的

近红外(near-infrared，NIR)图像在夜视和去雾等方面发挥着重要作用，RGB-NIR图像融合是一种常见且有效的处理方式。在实际图像处理过程中，图像的不同对象区域因特性不同需要进行差异化处理，但是现有图像融合算法在植被和天空图像区域存在明显不足。对此，提出RGB-NIR联合图像的植被和天空区域概率模板生成算法。

方法

以植被为感兴趣区域，基于RGB图像各通道比值和扩展归一化植被指数(normalized difference vegetation index，NDVI)两种特征，提出RGB-NIR联合图像的植被区域概率模板生成算法。以天空为感兴趣区域，基于透射率图引导的局部熵和扩展NDVI两种特征，结合像素高度信息，提出RGB-NIR联合图像的天空区域概率模板生成算法。两种算法生成的植被和天空的概率模板在RGB-NIR图像融合过程中利用概率模板对权重矩阵进行修正，可明显改善融合效果。

结果

检测植被的模板生成算法与传统NDVI进行比较，在对比度和鲁棒性方面有更大优势；与语义分割进行比较，在准确度和纹理细节上有更好表现。检测天空的模板生成算法与当前的概率模板天空检测算法相比，准确率更高，边缘过渡更平滑；与当前的二值模板天空检测算法相比，在检测效果相当的情况下能保留更多细节信息，并且对小物体的划分更为准确。以本文检测算法修正后的图像融合结果在保持细节增强效果的同时，视觉感观更为自然，在定量指标上也更占优势。

结论

本文提出的概率模板生成算法结果准确、性能鲁棒，能有效提升RGB-NIR图像融合的效果，特别是在涉及权重的图像融合中能更好地结合与应用。

Abstract

Objective

The imaging mechanism of near-infrared (NIR) images is different from that of visible images. It can receive the infrared radiation emitted by the object and convert it into grayscale values. Therefore

stronger infrared radiation in the scene yields higher grayscale value in the NIR image and its adaptability to harsh environments (e.g.

fog

haze) is better than the visible light imaging. To take advantage of NIR images

RGB-NIR image fusion is a common and effective processing method

which has been widely used in various image vision applications

including recognition

detection and surveillance. Multiple objects will have different imaging results in the same image in terms of their reflection and infrared radiation features

and the same object will have different appearances in visible and NIR images as well. For example

the vegetation part appears as low gray-scale values in the RGB image

but high gray-scale values in the NIR image. In addition

current image fusion algorithms have been challenging in specific regions like vegetation and sky. Therefore

an accurate and robust region detection method is necessary for regional-based processing. However

most algorithms are concerned of single image only and cannot meet the requirements for RGB-NIR image-fused region detection.

Method

We develop a probability-mask generation method from vegetation and sky regions based on RGB-NIR image pairs. The vegetation region: 1) to preserve high contrast and smooth transition

we obtain the ratio of multiple channels of RGB images with the extended normalized difference vegetation index (NDVI). 2) To avoid the extreme case that the red channel is of value minimum or maximum

we use the relationship between NIR and luminance instead of red channel. 3) To get the detection result

we integrate the ratio-guided and the NDVI-extended into the probability mask of vegetation. The sky-region: 1) the local-entropy feature of the RGB image is calculated and a transmission map is for guidance. 2) The guided feature and the extended NDVI is combined

and the results with the height of pixels is enhanced

according to the prior that the sky basically has a great probability of appearing in the upper part of the natural-scene image. 3) The result is a probability mask and considered as the sky detection result. The vegetation and sky detection based algorithms produce corresponding probability masks. We can incorporate them into RGB-NIR image fusion algorithms to improve image quality. The original algorithm uses the Laplacian-Gaussian pyramid and the weight map for multi-scale fusion. We modify the weight map of the NIR image by multiplying it with the vegetation and sky probability masks

and then replace the original NIR weight map with the modified one. The rest of the fusion algorithm remains unchanged.

Result

Our algorithm is evaluated on a public dataset containing outdoors images

including country

field

forest

indoor

mountain

old-building

street

urban

and water. To express the health of vegetation in the field of remote sensing

we compare the proposed vegetation detection algorithm with the traditional NDVI. The experimental results of image fusion indicate that our image fusion algorithm can perform better by incorporating the region masks both quantitatively and qualitatively

and produce more realistic and natural images perceptually. Moreover

we analyze the difference between the probability mask and the binary mask when applied to image fusion in the same way. The results show that selected probability mask makes the fused images more colorful and rich in details.

Conclusion

Our probability mask generation algorithm of vegetation and sky is potential to high accuracy and robustness. Specifically

the detected areas in result images are accurate with clear details and smooth transition

and small objects can be segmented properly. Moreover

our algorithm is beneficial to improving the performance of RGB-NIR image fusion

especially on weight map

making the results have enhanced details and natural colors. It is easy-used without complicated calculations. It is worthy note that our algorithm is more suitable for natural scenes generally.

关键词

Keywords

references

Awad M, Elliethy A and Aly H A. 2020. Adaptive near-infrared and visible fusion for fast image enhancement. IEEE Transactions on Computational Imaging, 6: 408-418[DOI: 10.1109/TCI.2019.2956873]

Brown M and Süsstrunk S. 2011. Multi-spectral SIFT for scene category recognition//Proceedings of 2011 Conference on Computer Vision and Pattern Recognition. Colorado Springs, USA: IEEE: 177-184[DOI: 10.1109/CVPR.2011.5995637 http://dx.doi.org/10.1109/CVPR.2011.5995637 ]

Connah D, Drew M S and Finlayson G D. 2014. Spectral edge image fusion: theory and applications//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 65-80[DOI: 10.1007/978-3-319-10602-1_5 http://dx.doi.org/10.1007/978-3-319-10602-1_5 ]

Fan J Y, Chen T and Lu S J. 2016. Vegetation coverage detection from very high resolution satellite imagery//Proceedings of 2015 Visual Communications and Image Processing. Singapore, Singapore: IEEE: 1-4[DOI: 10.1109/VCIP.2015.7457846 http://dx.doi.org/10.1109/VCIP.2015.7457846 ]

Han J and Bhanu B. 2007. Fusion of color and infrared video for moving human detection. Pattern Recognition, 40(6): 1771-1784[DOI: 10.1016/j.patcog.2006.11.010]

He K M, Sun J and Tang X O. 2011. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12): 2341-2353[DOI: 10.1109/TPAMI.2010.168]

Kim S E, Park T H and Eom I K. 2020. Fast single image dehazing using saturation based transmission map estimation. IEEE Transactions on Image Processing, 29: 1985-1998[DOI: 10.1109/TIP.2019.2948279]

Kriegler F J, Malila W A, Nalepka R F and Richardson W. 1969. Preprocessing transformations and their effect on multispectral recognition//The 6th International Symposium on Remote Sensing of Environment. Ann Arbor, USA: University of Michigan: 97-131

Kumar P, Mittal A and Kumar P. 2006. Fusion of thermal infrared and visible spectrum video for robust surveillance//Proceedings of the 5th Indian Conference on Computer Vision. Madurai, India: Springer: 528-539[DOI: 10.1007/11949619_47 http://dx.doi.org/10.1007/11949619_47 ]

Li Z, Hu H M, Zhang W, Pu S L and Li B. 2020. Spectrum characteristics preserved visible and near-infrared image fusion algorithm. IEEE Transactions on Multimedia, 23: 306-319[DOI: 10.1109/TMM.2020.2978640]

Qu G H, Zhang D L and Yan P F. 2002. Information measure for performance of image fusion. Electronics Letters, 38(7): 313-315[DOI: 10.1049/el:20020212]

Salazar-Colores S, Moya-Sánchez E U, Ramos-Arreguín J M, Cabal-Yépez E, Flores G and Cortés U. 2020. Fast single image defogging with robust sky detection. IEEE Access, 8: 149176-149189[DOI: 10.1109/ACCESS.2020.3015724]

Schaul L, Fredembach C and Süsstrunk S. 2009. Color image dehazing using the near-infrared//Proceedings of the 16th IEEE International Conference on Image Processing (ICIP). Cairo, Egypt: IEEE: 1629-1632[DOI: 10.1109/ICIP.2009.5413700 http://dx.doi.org/10.1109/ICIP.2009.5413700 ]

Singh R, Vatsa M and Noore A. 2008. Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition. Pattern Recognition, 41(3): 880-893[DOI: 10.1016/j.patcog.2007.06.022]

Süsstrunk S, Fredembach C and Tamburrino D. 2010. Automatic skin enhancement with visible and near-infrared image fusion//Proceedings of the 18th ACM International Conference on Multimedia. Firenze, Italy: Association for Computing Machinery: 1693-1696[DOI: 10.1145/1873951.1874324 http://dx.doi.org/10.1145/1873951.1874324 ]

Vanmali A V and Gadre V M. 2017. Visible and NIR image fusion using weight-map-guided Laplacian-Gaussian pyramid for improving scene visibility. Sādhanā, 42(7): 1063-1082[DOI: 10.1007/s12046-017-0673-1]

Wang J, Peng J Y, Feng X Y, He G Q and Fan J P. 2014. Fusion method for infrared and visible images by using non-negative sparse representation. Infrared Physics and Technology, 67: 477-489[DOI: 10.1016/j.infrared.2014.09.019]

Xiang T Z, Yan L and Gao R R. 2015. A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain. Infrared Physics and Technology, 69: 53-61[DOI: 10.1016/j.infrared.2015.01.002]

Xiao T T, Liu Y C, Zhou B, Jiang Y and Jian S. 2018. Unified perceptual parsing for scene understanding//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 432-448[DOI: 10.1007/978-3-030-01228-1_26 http://dx.doi.org/10.1007/978-3-030-01228-1_26 ]

Xydeas C S and Petrovic' V. 2000. Objective image fusion performance measure. Electronics Letters, 36(4): 308-309[DOI: 10.1049/el:20000267]

Yu X L, Ren J L, Chen Q and Sui X. 2014. A false color image fusion method based on multi-resolution color transfer in normalization YC B C R space. Optik, 125(20): 6010-6016[DOI: 10.1016/j.ijleo.2014.07.059]

Zafarifar B and de With P H N. 2008. Blue sky detection for picture quality enhancement//Proceedings of the 8th International Conference on Advanced Concepts for Intelligent Vision Systems. Antwerp, Belgium: Springer: 522-532[DOI: 10.1007/11864349_48 http://dx.doi.org/10.1007/11864349_48 ]

Zhang G M, Pan G F and Liu J X. 2020. Domain adaptation for semantic segmentation based on adaption learning rate. Journal of Image and Graphics, 25(5): 913-925

张桂梅, 潘国峰, 刘建新. 2020. 域自适应城市场景语义分割. 中国图象图形学报, 25(5): 913-925 [DOI: 10.11834/jig.190424]

Zhou B L, Zhao H, Puig X, Xiao T T, Fidler S, Barriuso A and Torralba A. 2019. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 127(3): 302-321[DOI:10.1007/s11263-018-1140-0]

Alert me when the article has been cited

提交

Information interaction linear Transformer network for infrared and visible image fusion

Multi-branch non-homogeneous image dehazing based on concentration partitioning and image fusion

Nighttime image dehazing based on transmittance correction and layered optimization

Survey of texture optimization algorithms for 3D reconstructed scenes

Deep network-interpreted multispectral image fusion in remote sensing