浏览全部资源
扫码关注微信
Published： 16 October 2023 ，
移动端阅览
陈浩东， 纪庆革. 2023. 用于行人轨迹预测的场景限制时空图卷积网络. 中国图象图形学报， 28(10):3163-3175
Chen Haodong， Ji Qingge. 2023. Scene-constrained spatial-temporal graph convolutional network for pedestrian trajectory prediction. Journal of Image and Graphics， 28(10):3163-3175
目的
2
针对行人轨迹预测问题，已有的几种结合场景信息的方法基于合并操作通过神经网络隐式学习场景与行人运动的关联，无法直观地解释场景对单个行人运动的调节作用。除此之外，基于图注意力机制的时空图神经网络旨在学习全局模式下行人之间的社会交互，在人群拥挤场景下精度不佳。鉴于此，本文提出一种场景限制时空图卷积神经网络（scene-constrained spatial-temporal graph convolutional neural network，Scene-STGCNN）。
方法
2
Scene-STGCNN由运动模块、基于场景的微调模块、时空卷积和时空外推卷积组成。运动模块以时空图卷积提取局部行人时空特征，避免了时空图神经网络在全局模式下学习交互的局限性。基于场景的微调模块将场景信息嵌入为掩模矩阵，用来调节运动模块生成的中间运动特征，具备实际场景下的物理解释性。通过最小化核密度估计下真实轨迹的负对数似然，增强Scene-STGCNN输出的多模态性，减少预测误差。
结果
2
实验在公开数据集ETH（包含ETH和HOTEL）和UCY（包含UNIV、ZARA1和ZARA2）上与其他7种主流方法进行比较，就平均值而言，相对于性能第2的模型，平均位移误差（average displacement error，ADE）值减少了12%，最终位移误差（final displacement error，FDE）值减少了9%。在同样的数据集上进行了消融实验以验证基于场景的微调模块的有效性，结果表明基于场景的微调模块能有效建模场景对行人轨迹的调节作用，从而减小算法的预测误差。
结论
2
本文提出的场景限制时空图卷积网络能有效融合场景和行人运动，在学习局部模式下行人交互的同时基于场景特征对轨迹特征做实时性调节，相比于其他主流方法，具有更优的性能。
Objective
2
Pedestrian trajectory prediction is essential for such domains like unmanned vehicles， security surveillance， and social robotics nowadays. Trajectory prediction is beneficial for computer systems to perform better decision making and planning to some extent. Current methods are focused on pedestrian trajectory information， and scene elements-related spatial constraints on pedestrian motion in the same space are challenged to explain human-to-human social interactions further， in which future location of pedestrians cannot be located in building walls， and pedestrians at building corners undergo large velocity direction deflections due to cornering behavior. The pathways can be focused on the integrated scene information， for which the scene image is melted into a one-dimensional vector and merged with the trajectory information. Two-dimensional spatial signal of the scene will be distorted and it cannot be intuitively explained according to the modulating effect of the scene on pedestrian motion. To build a spatiotemporal graph representation of pedestrians， recent graph neural network （GNN） is used to develop a method based on graph attention network （GAT）， in which pedestrians are as the graph nodes， trajectory features as the node attributes， and pedestrians-between spatial interactions are as the edges in the graph. These sorts of methods can be used to focus on pedestrians-between social interactions in the global scale. However， for crowded scenes， graph attention mechanism may not be able to assign appropriate weights to each pedestrian accurately， resulting in poor algorithm accuracy. To resolve the two problems mentioned above， we develop a scene constraints-based spatiotemporal graph convolutional network， called Scene-STGCNN， which aggregates pedestrian motion status with a graph convolutional neural network for local interactions， and it achieves accurate aggregation of pedestrian motion status with a small number of parameters. At the same time， we design a scene-based fine-tuning module to explicitly model the modulating effect of scenes on pedestrian motion with the information of neighboring scene changes as input.
Method
2
Scene-STGCNN consists of a motion module， a scene-based fine-tuning module， spatiotemporal convolution， and spatiotemporal extrapolation convolution. For the motion module， the graph convolution is a 1 × 1 core-sized convolutional neural network （CNN） layer for embedding pedestrian velocity information. The residual convolution is composed of CNN layer of 1 × 1 kernel size and BatchNorm （BN） layer. Temporal convolution is organized of BN layer， PReLU layer， 3 × 1 core-sized CNN layer， BN layer and Dropout layer as well. The motion module takes the pedestrian velocity spatiotemporal graph and the scene mask matrix as input， in which CNN-based pedestrian velocity spatiotemporal graph is encoded and the pedestrian spatiotemporal features of existing multiple frames are fused. For the scene-based fine-tuning module， temporal neighboring scene change information is first introduced to generate the scene-based pedestrian spatiotemporal map， and the embedding of the pedestrian spatiotemporal map by scene convolution is then performed to obtain the scene mask matrix， which is used to make Hadamard products with the intermediate motion features in the motion module. The real-time regulation role of the scene on pedestrians can be explicitly modeling further. Spatiotemporal convolution as a transition coding network consists of two temporal gating units and a spatial convolution， which is used to enhance the temporal correlation and contextual spatial dependence of pedestrian motion. A two-dimensional Gaussian distribution-related trajectory distribution is generated in terms of temporal extrapolation convolution. The kernel density estimation-based negative log-likelihood as the loss function will enhance the multimodality of the Scene-STGCNN prediction distribution while the prediction loss is optimized.
Result
2
Experiments are carried out to compare with the other related seven popular methods on the publicly available datasets ETH （including ETH and HOTEL） and UCY （including UNIV， ZARA1， and ZARA2）. The average displacement error （ADE） values are optimized by 12%， and the final displacement error （FDE） values are optimized by 9% in terms of average values. Ablation experiments are used to verify the effectiveness of the scene-based fine-tuning module， and the results demonstrate that the scene-based fine-tuning module can effectively model the modulation effect of the scene on pedestrian trajectory， and the prediction error of the algorithm is optimized as well. In addition， qualitative analysis is focused on the issues of Scene-STGCNN-captured inherent patterns of pedestrian motion and the involved prediction distribution. The visualization results show that Scene-STGCNN can be used to learn the pedestrian motion patterns effectively while maintaining accurate predictions.
Conclusion
2
we facilitate a pedestrian trajectory prediction model， called Scene-STGCNN， which can fuse scene information with trajectory features effectively through a scene-based fine-tuning module. Furthermore， Scene-STGCNN potentials can be focused on scene information-related pedestrian trajectory prediction method to a certain extent via modeling the modulation effect of scene on pedestrian motion.
行人轨迹预测场景空间限制时空特征提取时空卷积核密度估计
pedestrian trajectory predictionspatial constraints of scenespatio-temporal feature extractionspatio-temporal convolutionkernel density estimation
Alahi A， Goel K， Ramanathan V， Robicquet A， Li F F and Savarese S. 2016. Social LSTM： human trajectory prediction in crowded spaces//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 961-971 ［DOI： 10.1109/CVPR.2016.110http://dx.doi.org/10.1109/CVPR.2016.110］
Amirian J， Hayet J B and Pettré J. 2019. Social ways： learning multi-modal distributions of pedestrian trajectories with GANs//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Long Beach， USA： IEEE： 2964-2972 ［DOI： 10.1109/CVPRW.2019.00359http://dx.doi.org/10.1109/CVPRW.2019.00359］
Antonini G， Bierlaire M and Weber M. 2006. Discrete choice models of pedestrian walking behavior. Transportation Research Part B： Methodological， 40（8）： 667-687 ［DOI： 10.1016/j.trb.2005.09.006http://dx.doi.org/10.1016/j.trb.2005.09.006］
Chen G Y， Li J L， Lu J W and Zhou J. 2021. Human trajectory prediction via counterfactual analysis//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 9804-9813 ［DOI： 10.1109/ICCV48922.2021.00968http://dx.doi.org/10.1109/ICCV48922.2021.00968］
Dendorfer P， Elflein S and Leal-Taixé L. 2021a. MG-GAN： a multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 13138-13147 ［DOI： 10.1109/ICCV48922.2021.01291http://dx.doi.org/10.1109/ICCV48922.2021.01291］
Dendorfer P， Ošep A and Leal-Taixé L. 2021b. Goal-GAN： multimodal trajectory prediction based on goal position estimation//Proceedings of the 15th Asian Conference on Computer Vision. Kyoto， Japan： Springer： 405-420 ［DOI： 10.1007/978-3-030-69532-3_25http://dx.doi.org/10.1007/978-3-030-69532-3_25］
Gupta A， Johnson J， Li F F， Savarese S and Alahi A. 2018. Social GAN： socially acceptable trajectories with generative adversarial networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 2255-2264 ［DOI： 10.1109/CVPR.2018.00240http://dx.doi.org/10.1109/CVPR.2018.00240］
Helbing D and Molnr P. 1995. Social force model for pedestrian dynamics. Physical Review E， 51（5）： 4282-4286 ［DOI： 10.1103/PhysRevE.51.4282http://dx.doi.org/10.1103/PhysRevE.51.4282］
Huang Y F， Bi H K， Li Z X， Mao T L and Wang Z Q. 2019. STGAT： modeling spatial-temporal interactions for human trajectory prediction//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 6271-6280 ［DOI： 10.1109/ICCV.2019.00637http://dx.doi.org/10.1109/ICCV.2019.00637］
Ivanovic B and Pavone M. 2019. The trajectron： probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 2375-2384 ［DOI： 10.1109/ICCV.2019.00246http://dx.doi.org/10.1109/ICCV.2019.00246］
Kosaraju V， Sadeghian A， Martín-Martín R， Reid L， Rezatofighi S H and Savarese S. 2019. Social-BiGAT： multimodal trajectory forecasting using Bicycle-GAN and graph attention networks ［EB/OL］. ［2022-09-03］. https://arxiv.org/pdf/1907.03395.pdfhttps://arxiv.org/pdf/1907.03395.pdf
Lan H and Liu Q Y. 2020. Image generation from scene graph with graph attention network. Journal of Image and Graphics， 25（8）： 1591-1603
兰红， 刘秦邑. 2020. 图注意力网络的场景图到图像生成模型. 中国图象图形学报， 25（8）： 1591-1603 ［DOI： 10.11834/jig.190515http://dx.doi.org/10.11834/jig.190515］
Lea C， Flynn M D， Vidal R， Reiter A and Hager G D. 2017. Temporal convolutional networks for action segmentation and detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 1003-1012 ［DOI： 10.1109/CVPR.2017.113http://dx.doi.org/10.1109/CVPR.2017.113］
Lee N， Choi W， Vernaza P， Choy C B， Torr P H S and Chandraker M. 2017. DESIRE： distant future prediction in dynamic scenes with interacting agents//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 2165-2174 ［DOI： 10.1109/CVPR.2017.233http://dx.doi.org/10.1109/CVPR.2017.233］
Lerner A， Chrysanthou Y and Lischinski D. 2007. Crowds by example. Computer Graphics Forum， 26（3）： 655-664 ［DOI： 10.1111/j.1467-8659.2007.01089.xhttp://dx.doi.org/10.1111/j.1467-8659.2007.01089.x］
Liu Y B， Li H N， Zhang C Q， Xiao Z T， Zhang F， Wei Y， Gao Y Z， Shi F， Shan F and Shen D G. 2022. Diagnosis of COVID-19 by using structural attention graph neural network. Journal of Image and Graphics， 27（3）： 750-761
刘彦北， 李赫南， 张长青， 肖志涛， 张芳， 隗英， 高耀宗， 石峰， 单飞， 沈定刚. 2022. 结构图注意力网络的新冠肺炎轻重症诊断. 中国图象图形学报， 27（3）： 750-761 ［DOI： 10.11834/jig.210682http://dx.doi.org/10.11834/jig.210682］
Mangalam K， An Y， Girase H and Malik J. 2021. From goals， waypoints and paths to long term human trajectory forecasting//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 15213-15222 ［DOI： 10.1109/ICCV48922.2021.01495http://dx.doi.org/10.1109/ICCV48922.2021.01495］
Mangalam K， Girase H， Agarwal S， Lee K H， Adeli E， Malik J and Gaidon A. 2020. It is not the journey but the destination： endpoint conditioned trajectory prediction//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 759-776 ［DOI： 10.1007/978-3-030-58536-5_45http://dx.doi.org/10.1007/978-3-030-58536-5_45］
Mohamed A， Qian K， Elhoseiny M and Claudel C. 2020. Social-STGCNN： a social spatio-temporal graph convolutional neural network for human trajectory prediction//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 14412-14420 ［DOI： 10.1109/CVPR42600.2020.01443http://dx.doi.org/10.1109/CVPR42600.2020.01443］
Pellegrini S， Ess A and van Gool L. 2010. Improving data association by joint modeling of pedestrian trajectories and groupings//Proceedings of the 11th European Conference on Computer Vision. Heraklion， Greece： Springer： 452-465 ［DOI： 10.1007/978-3-642-15549-9_33http://dx.doi.org/10.1007/978-3-642-15549-9_33］
Sadeghian A， Kosaraju V， Sadeghian A， Hirose N， Rezatofighi H and Savarese S. 2019. SoPhie： an attentive GAN for predicting paths compliant to social and physical constraints//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 1349-1358 ［DOI： 10.1109/CVPR.2019.00144http://dx.doi.org/10.1109/CVPR.2019.00144］
Salzmann T， Ivanovic B， Chakravarty P and Pavone M. 2020. Trajectron++： dynamically-feasible trajectory forecasting with heterogeneous data//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 683-700 ［DOI： 10.1007/978-3-030-58523-5_40http://dx.doi.org/10.1007/978-3-030-58523-5_40］
Tay M K C and Laugier C. 2007. Modelling smooth paths using gaussian processes//Proceedings of the 6th International Conference on Field and Service Robotics. Chamonix， France： Springer： 381-390 ［DOI： 10.1007/978-3-540-75404-6_36http://dx.doi.org/10.1007/978-3-540-75404-6_36］
Xu Y Y， Piao Z and Gao S H. 2018. Encoding crowd interaction with deep neural network for pedestrian trajectory prediction//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 5275-5284 ［DOI： 10.1109/CVPR.2018.00553http://dx.doi.org/10.1109/CVPR.2018.00553］
Yamaguchi K， Berg A C， Ortiz L E and Berg T L. 2011. Who are you with and where are you going?//Proceeding of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Cdorado Springs，USA： IEEE： 1345-1352 ［DOI： 10.1109/CVPR.2011.5995468http://dx.doi.org/10.1109/CVPR.2011.5995468］
Yan S J， Xiong Y J and Lin D H. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition//Proceedings of the 32nd AAAI Conference on Artificial Intelligence and the 30th Innovative Applications of Artificial Intelligence Conference and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence. New Orleans， USA： AAAI Press： 7444-7452
Yu B， Yin H T and Zhu Z X. 2018. Spatio-temporal graph convolutional networks： a deep learning framework for traffic forecasting//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm， Sweden： IJCAI.org： 3634-3640 ［DOI： 10.24963/ijcai.2018/505http://dx.doi.org/10.24963/ijcai.2018/505］
Yu C J， Ma X， Ren J W， Zhao H Y and Yi S. 2020. Spatio-temporal graph Transformer networks for pedestrian trajectory prediction//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 507-523 ［DOI： 10.1007/978-3-030-58610-2_30http://dx.doi.org/10.1007/978-3-030-58610-2_30］
相关作者
相关机构
京公网安备11010802024621