红外—可见光跨模态的行人检测综述

别倩; 王晓; 徐新; 赵启军; 王正; 陈军; 胡瑞敏

doi:10.11834/jig.220670

综述 | 浏览量 : 0 下载量: 259 CSCD: 4

PDF
导出
分享
收藏
专辑

红外—可见光跨模态的行人检测综述
Visible-infrared cross-modal pedestrian detection： a summary
2023年28卷第5期页码：1287-1307
收稿：2022-07-07，

修回：2022-12-06，

纸质出版：2023-05-16
DOI： 10.11834/jig.220670
稿件说明：

移动端阅览

别倩，王晓，徐新，赵启军，王正，陈军，胡瑞敏. 2023. 红外—可见光跨模态的行人检测综述. 中国图象图形学报， 28(05):1287-1307 DOI： 10.11834/jig.220670.

Bie Qian， Wang Xiao， Xu Xin， Zhao Qijun， Wang Zheng， Chen Jun， Hu Ruimin. 2023. Visible-infrared cross-modal pedestrian detection： a summary. Journal of Image and Graphics， 28(05):1287-1307 DOI： 10.11834/jig.220670.

摘要

可见光图像在光照充足的条件下可以提供一系列辅助检测行人的信息，如颜色和纹理等信息，但在低照度场景下表现并不理想。红外图像虽然不能提供颜色和纹理信息，但红外图像根据热辐射差异成像而不依赖于光照条件这一特性，使其可以在低照度场景下有效区分行人区域与背景区域并提供清晰的行人轮廓信息。由于红外和可见光两种模态之间直观的互补性，同时使用红外和可见光图像的行人检测任务被认为是一个很有前景的研究方向，受到了广泛关注，大幅促进了在安防（如安全监控和自动驾驶）和疫情防控等领域应用的发展。本文对红外—可见光跨模态的行人检测工作进行全面梳理，并对未来方向进行深入思考。首先，该课题具有独特性质。可见光图像对应三通道的颜色信息而红外图像对应单通道的温差信息，如何在两种模态存在本质差异的前提下，充分利用二者的互补性是红外—可见光跨模态行人检测领域的核心挑战和主要任务。其次，近几年红外—可见光跨模态行人检测研究针对的问题可分为两类，即模态差异大和实际应用难。针对模态差异大的问题，可分为图像未对准和融合不充分两类问题。针对实际应用难的问题，又分为标注成本、实时检测和硬件成本3类问题。本文依次对跨模态行人检测的主要研究方向展开细致且全面的描述并进行相应的总结。然后，详细地介绍与跨模态行人检测相关的数据集和评价指标，并以不同的评价指标对相关方法在不同层面上进行比较。最后，对跨模态行人检测领域存在的且尚未解决的问题进行讨论，并提出对未来相关工作方向的一些思考。

Abstract

The precision of pedestrian detection is focused on instances-relevant location on given input images. However， due to the perception of visible images to light changes， visible images are challenged for lower visibility conditions like extreme weathers. Hence， visible images-based pedestrian detection is merely suitable for the development of temporal applications like autonomous driving and video surveillance. The infrared image can provide a clear pedestrian profile for such low-visibility scenes according to the temperature difference between the human body and the environment. Under the circumstances of sufficient light， visible images can also provide more information-lacked in infrared images like hair， face， and other related features. Visible and infrared images can provide visual information-added in common. However， the key challenges of visible and infrared images is to utilize the two modalities-between and their modality-specific noise mutually. To generate temperature information， the difference is leaked out that the visible image consists of color information in red， green， and blue （RGB） channels， while the infrared image has one channel only. And， imaging mechanism-based wavelength range of the two is different as well. The emerging deep learning technique based cross-modal pedestrian detection approaches have been developing dramatically. Our summary aims to review and analyze some popular researches on cross-modal pedestrian detection in recent years. It can be segmented into two categories： 1） the difference between two different modalities and 2） the cross-modal detectors application to the real scene. The application of cross-modal pedestrian detectors to the actual scene can be divided into three types： cost analysis-related data annotation， real-time detection， and cost-analysis of applications. The research aspects between two modalities can be divided into： the misalignment and the inadequate fusion. The misalignment of two modalities shows that the visible-infrared image pairs are required to be strictly aligned， and the features from different modalities are called to match at corresponding positions. The inadequate fusion of two modalities is required to maximize the mutual benefits between two modalities. The early research on the insufficient fusion of two-modality is related to the study of the fusion stage （when to fuse） of two-modality. The later studies on the insufficient fusion of two-modality data are focused on the study of the fusion methods （how to fuse） of two-modality. The fusion stage can be divided into three steps： image， feature， and decision. Similarly， the fusion methods can be segmented into three categories： image， feature， and detection. Subsequently， we introduce some commonly used cross-modal pedestrian detection datasets， including the Korea Advanced Institute of Science and Technology（KAIST）， the forward looking infrared radiometer（FLIR）， the computer vision center-14（CVC-14）， and the low-light visible-infrared parred（LLVIP）. Then， we introduce some evaluation metrics method for cross-modal pedestrian detectors， including missed rate （MR）， mean average precision （mAP）， and a pair of visible and thermal images in temporal （speed）. Finally， we summarize the challenges to be resolved in the field of cross-modal pedestrian detection and our predictions are focused on the future direction analysis of cross-modal pedestrian detection. 1） In the real world， due to the different parallax and field of view of two different sensors， the problem of misalignment of visible-infrared modality feature modules is more concerned about. However， the problem of unaligned modality features is possible to sacrifice the performance of the detector and hinder the use of unaligned data in datasets， and is not feasible to the application of dual sensors in real life to some extent. Thus， the problem of two modalities’ position is to be resolved as a key research direction. 2） At present， the datasets of cross-modal pedestrian detection are all captured on sunny days， and current advanced cross-modal pedestrian detection methods are only based on all-day pedestrian detection on sunny days. However， to realize the cross-modal pedestrian detection system throughout all day and all weathers， it is required to optimize and beyond day and night data on sunny days. We also need to focus on the data under extreme weather conditions. 3） Recent studies on cross-modal pedestrian detection are focused on datasets captured by vehicle-mounted cameras. Compared to datasets captured from the monitoring perspective， the scenes of vehicle-mounted datasets are changeable， which can suppress over-fitting effectively. However， the nighttime images in the vehicle-mounted datasets may be brighter than those of the surveillance perspective datasets because of their headlight brightness at night. Therefore， we predict that multiple visual-angles datasets can be used to train the cross-modal pedestrian detector at the same time. It can not only increase the robustness of the model in darker scenes， but also suppress over-fitting at a certain scene. 4） Autonomous driving systems and robot systems are required to be quick responded for detection results. Although many models have fast inference ability on GPU（graphics processing unit）， the inference speed on real devices need to be optimized， so real-time detection will be the continuous development direction of cross-modal pedestrian detection as well. 5） There is still a large gap in cross-modal pedestrian detection technology for small scale and partial or severe occluded pedestrians. However， driving systems-assisted detection and occlusion can be as a very common problem in life for small targets of pedestrians at a distance to alert drivers to slow down in advance. The cross-modal pedestrian detection technology can be forecasted and recognized for small scale targets and occlusion as the direction of future research.

关键词

Keywords

references

Andreas M ， Chris B ， Thomas B M ， Albert C and Sergio E . 2013 . Tri-modal person re-identification with RGB， depth and thermal features // Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition （CVPR） Workshops . Portland， USA ： IEEE： 301 - 307 ［ DOI： 10.1109/cvprw.2013.52 http://dx.doi.org/10.1109/cvprw.2013.52 ］

Beleznai C ， Steininger D ， Croonen G and Broneder E . 2018 . Multi-modal human detection from aerial views by fast shape-aware clustering and classification // Proceedings of the 10th IAPR Workshop on Pattern Recognition in Remote Sensing . Beijing， China ： IEEE： 1 - 6 ［ DOI： 10.1109/prrs.2018.8486236 http://dx.doi.org/10.1109/prrs.2018.8486236 ］

Blondel P ， Potelle A ， Pégard C and Lozano R . 2014 . Fast and viewpoint robust human detection in uncluttered environments // Proceedings of 2014 IEEE Visual Communications and Image Processing Conference . Valletta， Malta ： IEEE： 522 - 525 ［ DOI： 10.1109/vcip.2014.7051621 http://dx.doi.org/10.1109/vcip.2014.7051621 ］

Bondi E ， Jain R ， Aggrawal P ， Anand S ， Hannaford R ， Kapoor A ， Piavis J ， Shah S ， Joppa L ， Dilkina B and Tambe M . 2020 . BIRDSAI： a dataset for detection and tracking in aerial thermal infrared videos // Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass， USA ： IEEE： 1736 - 1745 ［ DOI： 10.1109/wacv45572.2020.9093284 http://dx.doi.org/10.1109/wacv45572.2020.9093284 ］

Bozcan I and Kayacan E . 2020 . AU-AIR： a multi-modal unmanned aerial vehicle dataset for low altitude traffic surveillance // Proceedings of 2020 IEEE International Conference on Robotics and Automation . Paris， France ： IEEE： 8504 - 8510 ［ DOI： 10.1109/ICRA40945.2020.9196845 http://dx.doi.org/10.1109/ICRA40945.2020.9196845 ］

Burt P J and Adelson E H . 1983 . The Laplacian pyramid as a compact image code //Fischler M A and Firschein O， eds. Readings in Computer Vision . Los Altos， USA ： Morgan Kaufmann： 671 - 679 ［ DOI： 10.1016/B978-0-08-051581-6.50065-9 http://dx.doi.org/10.1016/B978-0-08-051581-6.50065-9 ］

Candès E ， Demanet L ， Donoho D and Ying L X . 2006 . Fast discrete curvelet transforms . Multiscale Modeling and Simulation ， 5 （ 3 ）： 861 - 899 ［ DOI： 10.1137/05064182X http://dx.doi.org/10.1137/05064182X ］

Cao Y P ， Guan D Y ， Huang W L ， Yang J X ， Cao Y L and Qiao Y . 2019a . Pedestrian detection with unsupervised multispectral feature learning using deep neural networks . Information Fusion ， 46 ： 206 - 217 ［ DOI： 10.1016/j.inffus.2018.06.005 http://dx.doi.org/10.1016/j.inffus.2018.06.005 ］

Cao Y P ， Guan D Y ， Wu Y L ， Yang J X ， Cao Y L and Yang M Y . 2019b . Box-level segmentation supervised deep neural networks for accurate and real-time multispectral pedestrian detection . ISPRS Journal of Photogrammetry and Remote Sensing ， 150 ： 70 - 79 ［ DOI： 10.1016/j.isprsjprs.2019.02.005 http://dx.doi.org/10.1016/j.isprsjprs.2019.02.005 ］

Choi E J and Park D J . 2010 . Human detection using image fusion of thermal and visible image with new joint bilateral filter // Proceedings of the 5th International Conference on Computer Sciences and Convergence Information Technology . Seoul， Korea （South）： IEEE： 882 - 885 ［ DOI： 10.1109/iccit.2010.5711182 http://dx.doi.org/10.1109/iccit.2010.5711182 ］

Choi H ， Kim S ， Park K and Sohn K . 2016 . Multi-spectral pedestrian detection based on accumulated object proposal with fully convolutional networks // Proceedings of the 23rd International Conference on Pattern Recognition . Cancun， Mexico ： IEEE： 621 - 626 ［ DOI： 10.1109/icpr.2016.7899703 http://dx.doi.org/10.1109/icpr.2016.7899703 ］

Connah D ， Drew M S and Finlayson G D . 2015 . Spectral edge： gradient-preserving spectral mapping for image fusion . Journal of the Optical Society of America A ， 32 （ 12 ）： 2384 - 2396 ［ DOI： 10.1364/josaa.32.002384 http://dx.doi.org/10.1364/josaa.32.002384 ］

Davis J W and Sharma V . 2007 . Background-subtraction using contour-based fusion of thermal and visible imagery . Computer Vision and Image Understanding ， 106 （ 2/3 ）： 162 - 182 ［ DOI： 10.1016/j.cviu.2006.06.010 http://dx.doi.org/10.1016/j.cviu.2006.06.010 ］

de Oliveira D C and Wehrmeister M A . 2016 . Towards real-time people recognition on aerial imagery using convolutional neural networks // The 19th IEEE International Symposium on Real-Time Distributed Computing . York， UK ： IEEE： 27 - 34 ［ DOI： 10.1109/isorc.2016.14 http://dx.doi.org/10.1109/isorc.2016.14 ］

de Oliveira D C and Wehrmeister M A . 2018 . Using deep learning and low-cost RGB and thermal cameras to detect pedestrians in aerial images captured by multirotor UAV . Sensors ， 18 （ 7 ）： # 2244 ［ DOI： 10.3390/s18072244 http://dx.doi.org/10.3390/s18072244 ］

Deng Z and Latecki L J . 2017 . Amodal detection of 3D objects： inferring 3D bounding boxes from 2D ones in RGB-depth images // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： 398 - 406 ［ DOI： 10.1109/cvpr.2017.50 http://dx.doi.org/10.1109/cvpr.2017.50 ］

Dollar P ， Wojek C ， Schiele B and Perona P . 2012 . Pedestrian detection： an evaluation of the state of the art . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 34 （ 4 ）： 743 - 761 ［ DOI： 10.1109/tpami.2011.155 http://dx.doi.org/10.1109/tpami.2011.155 ］

Du D W ， Qi Y K ， Yu H Y ， Yang Y F ， Duan K W ， Li G R ， Zhang W G ， Huang Q M and Tian Q . 2018 . The unmanned aerial vehicle benchmark： object detection and tracking // Proceedings of the 15th European Conference on Computer Vision . Munich， Germany ： Springer： 375 - 391 ［ DOI： 10.1007/978-3-030-01249-6_23 http://dx.doi.org/10.1007/978-3-030-01249-6_23 ］

Fang Q Y and Wang Z K . 2022 . Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery . Pattern Recognition ， 130 ： # 108786 ［ DOI： 10.1016/j.patcog.2022.108786 http://dx.doi.org/10.1016/j.patcog.2022.108786 ］

Farooq A ， Jalal A and Kamal S . 2015 . Dense RGB-D map-based human tracking and activity recognition using skin joints features and self-organizing map . KSII Transactions on Internet and Information Systems ， 9 （ 5 ）： 1856 - 1869 ［ DOI： 10.3837/tiis.2015.05.017 http://dx.doi.org/10.3837/tiis.2015.05.017 ］

French G ， Finlayson G and Mackiewicz M . 2018 . Multi-spectral pedestrian detection via image fusion and deep neural networks . Journal of Imaging Science and Technology ， 62 （ 5 ）： # 50406 ［ DOI： 10.2352/J.ImagingSci.Technol.2018.62.5.050406 http://dx.doi.org/10.2352/J.ImagingSci.Technol.2018.62.5.050406 ］

F. A. Group . 2018 . Flir Thermal Dataset for Algorithm Training ［EB/OL］. ［ 2023-02-26 ］. https：//www.flir.in/oem/adas/adas-dataset-form/ https://www.flir.in/oem/adas/adas-dataset-form/ .

Golcarenarenji G ， Martinez-Alpiste I ， Wang Q and Alcaraz-Calero J M . 2021 . Efficient real-time human detection using unmanned aerial vehicles optical imagery . International Journal of Remote Sensing ， 42 （ 7 ）： 2440 - 2462 ［ DOI： 10.1080/01431161.2020.1862435 http://dx.doi.org/10.1080/01431161.2020.1862435 ］

Gonz􀅡lez A ， Fang Z J ， Socarras Y ， Serrat J ， Vázquez D ， Xu J L and López A M . 2016 . Pedestrian detection at day/night time with visible and fir cameras： a comparison . Sensors ， 16 （ 6 ）： # 820 ［ DOI： 10.3390/s16060820 http://dx.doi.org/10.3390/s16060820 ］

Guan D Y ， Cao Y P ， Yang J X ， Cao Y L and Tisse C L . 2018 . Exploiting fusion architectures for multispectral pedestrian detection and segmentation . Applied Optics ， 57 （ 18 ）： 108 - 116 ［ DOI： 10.1364/ao.57.00d108 http://dx.doi.org/10.1364/ao.57.00d108 ］

Guan D Y ， Cao Y P ， Yang J X ， Cao Y L and Yang M Y . 2019a . Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection . Information Fusion ， 50 ： 148 - 157 ［ DOI： 10.1016/j.inffus.2018.11.017 http://dx.doi.org/10.1016/j.inffus.2018.11.017 ］

Guan D Y ， Luo X ， Cao Y P ， Yang J X ， Cao Y L ， Vosselman G and Yang M Y . 2019b . Unsupervised domain adaptation for multispectral pedestrian detection // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Long Beach， USA ： IEEE： 434 - 443 ［ DOI： 10.1109/cvprw.2019.00057 http://dx.doi.org/10.1109/cvprw.2019.00057 ］

Gupta S ， Girshick R ， Arbeláez P and Malik J . 2014 . Learning rich features from RGB-D images for object detection and segmentation // Proceedings of the 13th European Conference on Computer Vision . Zurich， Switzerland ： Springer ， 345 - 360 ［ DOI： 10.1007/978-3-319-10584-0_23 http://dx.doi.org/10.1007/978-3-319-10584-0_23 ］

He R ， Wu X ， Sun Z A and Tan T N . 2017 . Learning invariant deep representation for NIR-VIS face recognition // Proceedings of the 31st AAAI Conference on Artificial Intelligence . San Francisco， USA ： AAAI： 2000 - 2006 ［ DOI： 10.5555/3298483.3298529 http://dx.doi.org/10.5555/3298483.3298529 ］

Hou Y L ， Song Y Y ， Hao X L ， Shen Y ， Qian M Y and Chen H J . 2018 . Multispectral pedestrian detection based on deep convolutional neural networks . Infrared Physics and Technology ， 94 ： 69 - 77 ［ DOI： 10.1016/j.infrared.2018.08.029 http://dx.doi.org/10.1016/j.infrared.2018.08.029 ］

Hsieh M R ， Lin Y L and Hsu W H . 2017 . Drone-based object counting by spatially regularized regional proposal network // Proceedings of 2017 IEEE International Conference on Computer Vision . Venice， Italy ： IEEE： 4165 - 4173 ［ DOI： 10.1109/iccv.2017.446 http://dx.doi.org/10.1109/iccv.2017.446 ］

Hwang S ， Park J ， Kim N ， Choi Y and Kweon I S . 2015 . Multispectral pedestrian detection： benchmark dataset and baseline // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston， USA ： IEEE： 1037 - 1045 ［ DOI： 10.1109/CVPR.2015.7298706 http://dx.doi.org/10.1109/CVPR.2015.7298706 ］

Jia X Y ， Zhu C ， Li M Z ， Tang W Q and Zhou W L . 2021 . LLVIP： a visible-infrared paired dataset for low-light vision // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops . Montreal， Canada ： IEEE： 3489 - 3497 ［ DOI： 10.1109/iccvw54120.2021.00389 http://dx.doi.org/10.1109/iccvw54120.2021.00389 ］

Jiang T T ， Liu Y ， Ma X and Sun J L . 2021 . Multi-path collaborative salient object detection based on RGB-T images . Journal of Image and Graphics ， 26 （ 10 ）： 2388 - 2399

蒋亭亭，刘昱，马欣，孙景林 . 2021 . 多支路协同的RGB-T图像显著性目标检测 . 中国图象图形学报， 26 （ 10 ）： 2388 - 2399 ［ DOI： 10.11834/jig.200317 http://dx.doi.org/10.11834/jig.200317 ］

Kim J U ， Park S and Ro Y M . 2022a . Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection . IEEE Transactions on Circuits and Systems for Video Technology ， 32 （ 3 ）： 1510 - 1523 ［ DOI： 10.1109/tcsvt.2021.3076466 http://dx.doi.org/10.1109/tcsvt.2021.3076466 ］

Kim J U ， Park S and Ro Y M . 2022b . Towards versatile pedestrian detector with multisensory-matching and multispectral recalling memory //Proceedings of the 36th AAAI Conference on Artificial Intelligence. ［s.l.］： AAAI： 1157 - 1165 ［ DOI： 10.1609/aaai.v36i1.20001 http://dx.doi.org/10.1609/aaai.v36i1.20001 ］

König D ， Adam M ， Jarvers C ， Layher G ， Neumann H and Teutsch M . 2017 . Fully convolutional region proposal networks for multispectral person detection // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Honolulu， USA ： IEEE： 243 - 250 ［ DOI： 10.1109/cvprw.2017.36 http://dx.doi.org/10.1109/cvprw.2017.36 ］

Kruthiventi S S S ， Sahay P and Biswal R . 2017 . Low-light pedestrian detection from RGB images using multi-modal knowledge distillation // Proceedings of 2017 IEEE International Conference on Image Processing . Beijing， China ： IEEE： 4207 - 4211 ［ DOI： 10.1109/icip.2017.8297075 http://dx.doi.org/10.1109/icip.2017.8297075 ］

Kumar S V A ， Yaghoubi E ， Das A ， Harish B S and Proença H . 2021 . The P-DESTRE： a fully annotated dataset for pedestrian detection， tracking， and short/long-term re-identification from aerial devices . IEEE Transactions on Information Forensics and Security ， 16 ： 1696 - 1708 ［ DOI： 10.1109/tifs.2020.3040881 http://dx.doi.org/10.1109/tifs.2020.3040881 ］

Lee Y ， Bui T D and Shin J . 2018 . Pedestrian detection based on deep fusion network using feature correlation // Proceedings of 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference . Honolulu， USA ： IEEE： 694 - 699 ［ DOI： 10.23919/apsipa.2018.8659688 http://dx.doi.org/10.23919/apsipa.2018.8659688 ］

Lee Y ， Chen J C ， Tseng C W and Lai S H . 2016 . Accurate and robust face recognition from RGB-D images with a deep learning approach // Proceedings of 2016 British Machine Vision Conference . York， UK ： BMVA Press： #3 ［ DOI： 10.5244/c.30.123 http://dx.doi.org/10.5244/c.30.123 ］

Leykin A ， Ran Y and Hammoud R . 2007 . Thermal-visible video fusion for moving target tracking and pedestrian classification // Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition . Minneapolis， USA ： IEEE： 1 - 8 ［ DOI： 10.1109/cvpr.2007.383444 http://dx.doi.org/10.1109/cvpr.2007.383444 ］

Li C Y ， Song D ， Tong R F and Tang M . 2018 . Multispectral pedestrian detection via simultaneous detection and segmentation // Proceedings of British Machine Vision Conference 2018 . Newcastle， UK ： BMVA Press： #225

Li C Y ， Song D ， Tong R F and Tang M . 2019 . Illumination-aware faster R-CNN for robust multispectral pedestrian detection . Pattern Recognition ， 85 ： 161 - 171 ［ DOI： 10.1016/j.patcog.2018.08.005 http://dx.doi.org/10.1016/j.patcog.2018.08.005 ］

Li D G ， Wei X ， Hong X P and Gong Y H . 2020a . Infrared-visible cross-modal person re-identification with an X modality // Proceedings of the 34th AAAI Conference on Artificial Intelligence . Palo Alto， USA ： AAAI： 4610 - 4617 ［ DOI： 10.1609/aaai.v34i04.5891 http://dx.doi.org/10.1609/aaai.v34i04.5891 ］

Li G F ， Yang Y F and Qu X D . 2020b . Deep learning approaches on pedestrian detection in hazy weather . IEEE Transactions on Industrial Electronics ， 67 （ 10 ）： 8889 - 8899 ［ DOI： 10.1109/tie.2019.2945295 http://dx.doi.org/10.1109/tie.2019.2945295 ］

Li J H ， Zhang P ， Wang X W and Huang S Z . 2020 . Infrared small-target detection algorithms： a survey . Journal of Image and Graphics ， 25 （ 9 ）： 1739 - 1753

李俊宏，张萍，王晓玮，黄世泽 . 2020 . 红外弱小目标检测算法综述 . 中国图象图形学报， 25 （ 9 ）： 1739 - 1753 ［ DOI： 10.11834/jig.190574 http://dx.doi.org/10.11834/jig.190574 ］

Li Y T . 2021 . Multispectral pedestrian detection in autonomous driving： a review . IEIE Transactions on Smart Processing and Computing ， 10 （ 1 ）： 10 - 16 ［ DOI： 10.5573/ieiespc.2021.10.1.010 http://dx.doi.org/10.5573/ieiespc.2021.10.1.010 ］

Lin C Z ， Lu J W ， Wang G and Zhou J . 2018 . Graininess-aware deep feature learning for pedestrian detection // Proceedings of the 15th European Conference on Computer Vision . Munich， Germany ： Springer： 745 - 761 ［ DOI： 10.1007/978-3-030-01240-3_45 http://dx.doi.org/10.1007/978-3-030-01240-3_45 ］

Liu J J ， Zhang S T ， Wang S and Metaxas N D . 2016a . Multispectral deep neural networks for pedestrian detection // Proceedings of British Machine Vision Conference 2016 . York， UK ： BMVA Press： 73.1- 73 .13

Liu T S ， Lam K M ， Zhao R and Qiu G P . 2022 . Deep cross-modal representation learning and distillation for illumination-invariant pedestrian detection . IEEE Transactions on Circuits and Systems for Video Technology ， 32 （ 1 ）： 315 - 329 ［ DOI： 10.1109/tcsvt.2021.3060162 http://dx.doi.org/10.1109/tcsvt.2021.3060162 ］

Liu W ， Anguelov D ， Erhan D ， Szegedy C ， Reed S ， Fu C Y and Berg A C . 2016b . SSD： single shot multiBox detector // Proceedings of the 14th European Conference on Computer Vision . Amsterdam， the Netherlands ： Springer： 21 - 37 ［ DOI： 10.1007/978-3-319-46448-0_2 http://dx.doi.org/10.1007/978-3-319-46448-0_2 ］

Liu W ， Liao S C ， Ren W Q ， Hu W D and Yu Y N . 2019 . High-level semantic feature detection： a new perspective for pedestrian detection // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 5182 - 5191 ［ DOI： 10.1109/cvpr.2019.00533 http://dx.doi.org/10.1109/cvpr.2019.00533 ］

Liu W J ， Dong L B and Qu H C . 2021 . Small-scale pedestrian detection based on improved R-FCN model . Journal of Image and Graphics ， 26 （ 10 ）： 2400 - 2410

刘万军，董利兵，曲海成 . 2021 . 改进R-FCN模型的小尺度行人检测 . 中国图象图形学报， 26 （ 10 ）： 2400 - 2410 ［ DOI： 10.11834/jig.200287 http://dx.doi.org/10.11834/jig.200287 ］

Lyu C ， Heyer P ， Munir A ， Platisa L ， Micheloni C ， Goossens B and Philips W . 2021 . Visible-thermal pedestrian detection via unsupervised transfer learning // Proceedings of the 5th International Conference on Innovation in Artificial Intelligence . Xiamen， China ： ACM： 158 - 163 ［ DOI： 10.1145/3461353.3461369 http://dx.doi.org/10.1145/3461353.3461369 ］

Ouyang W L and Wang X G . 2013 . Joint deep learning for pedestrian detection // Proceedings of 2013 IEEE International Conference on Computer Vision . Sydney， Australia ： IEEE： 2056 - 2063 ［ DOI： 10.1109/iccv.2013.257 http://dx.doi.org/10.1109/iccv.2013.257 ］

Park K ， Kim S and Sohn K . 2018 . Unified multi-spectral pedestrian detection based on probabilistic fusion networks . Pattern Recognition ， 80 ： 143 - 155 ［ DOI： 10.1016/j.patcog.2018.03.007 http://dx.doi.org/10.1016/j.patcog.2018.03.007 ］

Ranchin T and Wald L . 1993 . The wavelet transform for the analysis of remotely sensed images . International Journal of Remote Sensing ， 14 （ 3 ）： 615 - 619 ［ DOI： 10.1080/01431169308904362 http://dx.doi.org/10.1080/01431169308904362 ］

Redmon J and Farhadi A . 2017 . YOLO9000： better， faster， stronger // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 6517 - 6525 ［ DOI： 10.1109/cvpr.2017.690 http://dx.doi.org/10.1109/cvpr.2017.690 ］

Redmon J and Farhadi A . 2018 . YOLOv3： an incremental improvement ［EB/OL］. ［ 2018-04-08 ］. https：//arxiv.org/pdf/1804.02767.pdf https://arxiv.org/pdf/1804.02767.pdf

San-Biagio M ， Crocco M ， Cristani M ， Martelli S and Murino V . 2012 . Low-level multimodal integration on Riemannian manifolds for automatic pedestrian detection // Proceedings of the 15th International Conference on Information Fusion . Singapore ， Singapore ： IEEE： 2223 - 2229

Shao Y H ， Zhang X P ， Chu H Y ， Zhang X Q ， Zhang D and Rao Y B . 2022 . AIR-YOLOv3： aerial infrared pedestrian detection via an improved YOLOv3 with network pruning . Applied Sciences ， 12 （ 7 ）： # 3627 ［ DOI： 10.3390/app12073627 http://dx.doi.org/10.3390/app12073627 ］

Shao Z F ， Cheng G ， Ma J Y ， Wang Z Y ， Wang J M and Li D R . 2021 . Real-time and accurate UAV pedestrian detection for social distancing monitoring in COVID-19 pandemic . IEEE Transactions on Multimedia ， 24 ： 2069 - 2083 ［ DOI： 10.1109/TMM.2021.3075566 http://dx.doi.org/10.1109/TMM.2021.3075566 ］

Song S R and Xiao J X . 2016 . Deep sliding shapes for Amodal 3D object detection in RGB-D images // Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas， USA ： IEEE： 808 - 816 ［ DOI： 10.1109/cvpr.2016.94 http://dx.doi.org/10.1109/cvpr.2016.94 ］

Sun Y M ， Cao B ， Zhu P F and Hu Q H . 2022 . Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning . IEEE Transactions on Circuits sand Systems for Video Technology ， 32 （ 10 ）： 6700 - 6713 ［ DOI： 10.1109/TCSVT.2022.3168279 http://dx.doi.org/10.1109/TCSVT.2022.3168279 ］

Takumi K ， Watanabe K ， Ha Q S ， Tejero-De-Pablos A ， Ushiku Y and Harada T . 2017 . Multispectral object detection for autonomous vehicles // Proceedings of on Thematic Workshops of ACM Multimedia 2017 . California， USA ： ACM： 35 - 43 ［ DOI： 10.1145/3126686.3126727 http://dx.doi.org/10.1145/3126686.3126727 ］

Tian Y L ， Luo P ， Wang X G and Tang X O . 2015 . Deep learning strong parts for pedestrian detection // Proceedings of 2015 IEEE International Conference on Computer Vision . Santiago， USA ： IEEE： 1904 - 1912 ［ DOI： 10.1109/iccv.2015.221 http://dx.doi.org/10.1109/iccv.2015.221 ］

Vandersteegen M ， van Beeck K and Goedemé T . 2018 . Real-time multispectral pedestrian detection with a single-pass deep neural network // Proceedings of the 15th International Conference on Image Analysis and Recognition . Póvoa de Varzim， Portugal ： Springer： 419 - 426 ［ DOI： 10.1007/978-3-319-93000-8_47 http://dx.doi.org/10.1007/978-3-319-93000-8_47 ］

Vladimir V K ， Vladimir A K ， Jiri H ， Walter G K and Vladimir M . 2019 . Thermal GAN： multimodal color-to-thermal image translation for person re-identification in multispectral dataset . Computer Vision ECCV 2018 Workshops ， 606 - 624 ［ DOI：10.1007/978-3-030-11024-6_46］

Wagner J ， Fischer V ， Herman M and Behnke S . 2016 . Multispectral pedestrian detection using deep fusion convolutional neural networks // The 24th European Symposium on Artificial Neural Networks . Bruges， Belgium ： ESANN： 509 - 514

Wanchaitanawong N ， Tanaka M ， Shibata T and Okutomi M . 2021 . Multi-modal pedestrian detection with large misalignment based on modal-wise regression and multi-modal IoU // Proceedings of the 17th International Conference on Machine Vision and Applications . Aichi， Japan ： IEEE： 1 - 6 ［ DOI： 10.23919/mva51890.2021.9511366 http://dx.doi.org/10.23919/mva51890.2021.9511366 ］

Wang C Q ， Luo D ， Liu Y ， Xu B and Zhou Y J . 2022 . Near-surface pedestrian detection method based on deep learning for UAVs in low illumination environments . Optical Engineering ， 61 （ 2 ）： # 023103 ［ DOI： 10.1117/1.OE.61.2.023103 http://dx.doi.org/10.1117/1.OE.61.2.023103 ］

Wu A C ， Lin C Z and Zheng W S . 2022 . Single-modality self-supervised information mining for cross-modality person re-identification . Journal of Image and Graphics ， 27 （ 10 ）： 2843 - 2859

吴岸聪，林城梽，郑伟诗 . 2022 . 面向跨模态行人重识别的单模态自监督信息挖掘 . 中国图象图形学报， 27 （ 10 ）： 2843 - 2859 ［ DOI： 10.11834/jig.211050 http://dx.doi.org/10.11834/jig.211050 ］

Wu F ， Jing X Y ， Feng Y J ， Ji Y M and Wang R C . 2021 . Spectrum-aware discriminative deep feature learning for multi-spectral face recognition . Pattern Recognition ， 111 ： # 107632 ［ DOI： 10.1016/j.patcog.2020.107632 http://dx.doi.org/10.1016/j.patcog.2020.107632 ］

Xu D ， Ouyang W L ， Ricci E ， Wang X G and Sebe N . 2017 . Learning cross-modal deep representations for robust pedestrian detection // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 4236 - 4244 ［ DOI： 10.1109/cvpr.2017.451 http://dx.doi.org/10.1109/cvpr.2017.451 ］

Xu D F ， Anguelov D and Jain A . 2018 . Pointfusion： deep sensor fusion for 3D bounding box estimation // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 244 - 253 ［ DOI： 10.1109/cvpr.2018.00033 http://dx.doi.org/10.1109/cvpr.2018.00033 ］

Xu M ， Yu X S ， Chen D Y ， Wu C D ， Jia T and Ru J Y . 2018 . Pedestrian detection in complex thermal infrared surveillance scene . Journal of Image and Graphics ， 23 （ 12 ）： 1829 - 1837

许茗，于晓升，陈东岳，吴成东，贾同，茹敬雨 . 2018 . 复杂热红外监控场景下行人检测 . 中国图象图形学报， 23 （ 12 ）： 1829 - 1837 ［ DOI： 10.11834/jig.180299 http://dx.doi.org/10.11834/jig.180299 ］

Xu X W ， Zhang X Y ， Yu B ， Hu X S ， Rowen C ， Hu J T and Shi Y Y . 2019 . DAC-SDC low power object detection challenge for UAV applications . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 43 （ 2 ）： 392 - 403 ［ DOI： 10.1109/tpami.2019.2932429 http://dx.doi.org/10.1109/tpami.2019.2932429 ］

Yang F ， Fan H ， Chu P ， Blasch E and Ling H B . 2019 . Clustered object detection in aerial images // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul， Korea （South）： IEEE： 8311 - 8320 ［ DOI： 10.1109/iccv.2019.00840 http://dx.doi.org/10.1109/iccv.2019.00840 ］

Yang X X ， Qian Y Q ， Zhu H J ， Wang C X and Yang M . 2022 . BAANet： learning bi-directional adaptive attention gates for multispectral pedestrian detection // Proceedings of 2022 International Conference on Robotics and Automation . Philadelphia， USA ： IEEE： 2920 - 2926 ［ DOI： 10.1109/ICRA46639.2022.9811999 http://dx.doi.org/10.1109/ICRA46639.2022.9811999 ］

Ye M ， Lan X Y ， Li J W and Yuen P . 2018 . Hierarchical discriminative learning for visible thermal person re-identification // Proceedings of the 32nd AAAI Conference on Artificial Intelligence . New Orleans， USA ： AAAI： 7501 - 7508 ［ DOI： 10.1609/aaai.v32i1.12293 http://dx.doi.org/10.1609/aaai.v32i1.12293 ］

Zhang H ， Fromont E ， Lefevre S and Avignon B . 2020a . Multispectral fusion for object detection with cyclic fuse-and-refine blocks // Proceedings of 2020 IEEE International Conference on Image Processing . Abu Dhabi， United Arab Emirates ： IEEE： 276 - 280 ［ DOI： 10.1109/icip40778.2020.9191080 http://dx.doi.org/10.1109/icip40778.2020.9191080 ］

Zhang H ， Fromont E ， Lefevre S and Avignon B . 2021a . Guided attentive feature fusion for multispectral pedestrian detection // Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision . Waikoloa， USA ： IEEE： 72 - 80 ［ DOI： 10.1109/wacv48630.2021.00012 http://dx.doi.org/10.1109/wacv48630.2021.00012 ］

Zhang H ， Fromont E ， Lefevre S and Avignon B . 2021b . Deep active learning from multispectral data through cross-modality prediction inconsistency // Proceedings of 2021 IEEE International Conference on Image Processing . Anchorage， USA ： IEEE： 449 - 453 ［ DOI： 10.1109/icip42928.2021.9506322 http://dx.doi.org/10.1109/icip42928.2021.9506322 ］

Zhang H J ， Sun M S ， Li Q ， Liu L L ， Liu M and Ji Y Z . 2021c . An empirical study of multi-scale object detection in high resolution UAV images . Neurocomputing ， 421 ： 173 - 182 ［ DOI： 10.1016/j.neucom.2020.08.074 http://dx.doi.org/10.1016/j.neucom.2020.08.074 ］

Zhang L ， Liu Z Y ， Zhang S F ， Yang X ， Qiao H ， Huang K Z and Hussain A . 2019b . Cross-modality interactive attention network for multispectral pedestrian detection . Information Fusion ， 50 ： 20 - 29 ［ DOI： 10.1016/j.inffus.2018.09.015 http://dx.doi.org/10.1016/j.inffus.2018.09.015 ］

Zhang L ， Zhu X Y ， Chen X Y ， Yang X ， Lei Z and Liu Z Y . 2019a . Weakly aligned cross-modal learning for multispectral pedestrian detection // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul， Korea （South）： IEEE： 5126 - 5136 ［ DOI： 10.1109/ICCV.2019.00523 http://dx.doi.org/10.1109/ICCV.2019.00523 ］

Zhang Y T ， Yin Z S ， Nie L Z and Huang S . 2020b . Attention based multi-layer fusion of multispectral images for pedestrian detection . IEEE Access ， 8 ： 165071 - 165084 ［ DOI： 10.1109/access.2020.3022623 http://dx.doi.org/10.1109/access.2020.3022623 ］

Zhao C R ， Qi D ， Dou S G ， Tu Y P ， Sun T L ， Bai S ， Jiang X Y ， Bai X and Miao D Q . 2021 . Key technology for intelligent video surveillance： a review of person re-identiﬁcation . Science in China （Information Sciences）， 51 （ 12 ）： 1979 - 2015

赵才荣，齐鼎，窦曙光，涂远鹏，孙添力，柏松，蒋忻洋，白翔，苗夺谦 . 2021 . 智能视频监控关键技术：行人再识别研究综述 . 中国科学：信息科学）， 51 （ 12 ）： 1979 - 2015 ［ DOI： 10.1360/SSI-2021-021 http://dx.doi.org/10.1360/SSI-2021-021 ］

Zhao Y Q ， Rao Y ， Dong S P and Zhang J Y . 2020 . Survey on deep learning object detection . Journal of Image and Graphics ， 25 （ 4 ）： 629 - 654

赵永强，饶元，董世鹏，张君毅 . 2020 . 深度学习目标检测方法综述 . 中国图象图形学报， 25 （ 4 ）： 629 - 654 ［ DOI： 10.11834/jig.190307 http://dx.doi.org/10.11834/jig.190307 ］

Zhen Y ， Wang Z L and Wu F . 2021 . Pedestrian detection method based on density and score refinement . Journal of Image and Graphics ， 26 （ 2 ）： 425 - 437

甄烨，王子磊，吴枫 . 2021 . 融合密度和精细分数的行人检测 . 中国图象图形学报， 26 （ 2 ）： 425 - 437 ［ DOI： 10.11834/jig.200060 http://dx.doi.org/10.11834/jig.200060 ］

Zheng W S and Wu A C . 2018 . Asymmetric person re-identification： cross-view person tracking in a large camera network . Scientia Sinica Informationis ， 48 （ 5 ）： 545 - 563

郑伟诗，吴岸聪 . 2018 . 非对称行人重识别：跨摄像机持续行人追踪 . 中国科学：信息科学）， 48 （ 5 ）： 545 - 563 ［ DOI： 10.1360/N112018-00017 http://dx.doi.org/10.1360/N112018-00017 ］

Zheng Y ， Izzat I H and Ziaee S . 2019 . GFD-SSD： gated fusion double SSD for multispectral pedestrian detection ［EB/OL］. ［ 2019-03-16 ］. https：//arxiv.org/ftp/arxiv/papers/1903/1903.06999.pdf https://arxiv.org/ftp/arxiv/papers/1903/1903.06999.pdf

Zhou K L ， Chen L S and Cao X . 2020 . Improving multispectral pedestrian detection by addressing modality imbalance problems // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 787 - 803 ［ DOI： 10.1007/978-3-030-58523-5_46 http://dx.doi.org/10.1007/978-3-030-58523-5_46 ］

Zhu P F ， Wen L Y ， Bian X ， Ling H B and Hu Q H . 2018 . Vision meets drones： a challenge ［EB/OL］. ［ 2020-04-20 ］. https：//arxiv.org/pdf/1804.07437.pdf https://arxiv.org/pdf/1804.07437.pdf

Zhu P F ， Wen L Y ， Du D W ， Bian X ， Hu Q H and Ling H B . 2020 . Vision meets drones： past， present and future ［EB/OL］. ［ 2020-01-16 ］. https：//arxiv.org/pdf/2001.06303v1.pdf https://arxiv.org/pdf/2001.06303v1.pdf

Zhuang Y F ， Pu Z Y ， Hu J and Wang Y H . 2022 . Illumination and temperature-aware multispectral networks for edge-computing-enabled pedestrian detection . IEEE Transactions on Network Science and Engineering ， 9 （ 3 ）： 1282 - 1295 ［ DOI： 10.1109/tnse.2021.3139335 http://dx.doi.org/10.1109/tnse.2021.3139335 ］