浏览全部资源
扫码关注微信
Published： 16 July 2023 ，
移动端阅览
潘小鹍， 刘浩敏， 方铭， 王政， 张涌， 章国锋. 2023. 基于语义概率预测的动态场景单目视觉SLAM. 中国图象图形学报， 28(07):2151-2166
Pan Xiaokun， Liu Haomin， Fang Ming， Wang Zheng， Zhang Yong， Zhang Guofeng. 2023. Dynamic 3D scenario-oriented monocular SLAM based on semantic probability prediction. Journal of Image and Graphics， 28(07):2151-2166
目的
2
基于视觉的同步定位与建图（visual-based simultaneous localization and mapping，vSLAM）是计算机视觉以及机器人领域中的关键技术，其通过对输入的图像进行处理分析来感知周围的3维环境以及进行自身的定位。现有的SLAM系统大多依赖静态世界假设，在真实环境中的动态物体会严重影响视觉SLAM系统的稳定运行。同时，场景中静止与运动部分往往和其语义有密切关系，因而可以借助场景中的语义信息来提升视觉SLAM系统在动态环境下的稳定性。为此，提出一种新的基于语义概率预测的面向动态场景的单目视觉SLAM算法。
方法
2
结合语义分割的结果以及鲁棒性估计算法，通过对分割进行数据关联、状态检测，从概率的角度来表示观测的静止/运动状态，剔除动态物体上的观测对相机位姿估计的干扰，同时借助运动概率及时剔除失效的地图点，使系统在复杂动态的场景中依然能够稳定运行。
结果
2
在本文构建的复杂动态场景数据集上，提出的方法在跟踪精度和完整度上都显著优于现有的单目视觉SLAM方法，而且在TUM-RGBD数据集中的多个高动态序列上也取得了更好的结果。此外，本文定性比较了动态场景下的建图质量以及AR（augmented reality）效果。结果表明，本文方法明显优于对比方法。
结论
2
本文通过结合语义分割信息以及鲁棒性估计算法，对分割区域进行数据关联以及运动状态检测，以概率的形式表示2D观测的运动状态，同时及时剔除失效地图点，使相机位姿估计的精度以及建图质量有了明显提升，有效提高了单目视觉SLAM在高度动态环境中运行的鲁棒性。
Objective
2
Visual-based simultaneous localization and mapping （vSLAM） is essential for computer vision and robotic-related domain. The multiview and 3D structure scenarios can be recovered in terms of the input images analysis. Due to 3D objects in the real environment will seriously affect the stability of the vSLAM system， most of the existing vSLAM systems rely on static-scenarios assumption， which limits the application in the dynamic environment. Current geometry-based methods are focused on the negative effect alleviation of dynamic objects in checking some geometric constraints in 3D vision like epipolar constraints and re-projection error. Recent deep learning based semantic segmentation technology has been facilitating more effective information for SLAM system because the static and dynamic parts in the scene are often closely related to their semantics. Theoretically， due to the image information is transferred from pixel-level to semantic-level， the vSLAM system can be run stably in the dynamic environment in terms of the semantic information. However， some semantic-based SLAM schemes directly remove the semantic objects in the scene based on the segmentation results without considering their motion states. This may remove the areas that can provide stable visual features in some common real scenes， and the lack of sufficient observation will affect the stability of SLAM system severely. A feasible path is oriented to analyze the motion state of the semantic clustering， and applicability strategies are then implemented to alleviate the influence of the moving objects in visual localization module in vSLAM. The challenges of updating map in dynamic scene is required to be resolved as well.
Method
2
A new monocular vSLAM algorithm-based semantic probability prediction method is developed， which can combine semantic segmentation and robust estimation algorithm. To simplify training and generalization of network， the learning-based semantic segmentation module is opted. First， to track the certain cluster in 2D image space and keep the temporal consistency of segmentation， the input image data is segmented in relevance to segmentation results of existing frames. Next， robust estimation method and geometric constraints are used to detect the state of these clusters， which can be embedded with temporal consistency information. To determine whether the cluster is dynamic or not， the negative effect is alleviated by sacrifice the possible moving objects in the scene straightforward， which may lead to the reduction of the number of observations， further weaken the robustness of the system. Finally， the cluster state is used to represent probability-related the static/dynamic state of the 2D observation， which can melt the uncertainty of the edge of the dynamic object into the image space and the interference of the observation on the dynamic object can be reduced for the pose estimation of the camera. In addition to eliminating the moving observations， a spatially consistent sparse map is in consistent and beneficial for the stability of monocular vSLAM in dynamic scenes. Dynamic 3D objects in the scene will introduce some invalid map points into the map， resulting in the inconsistency between the map and the 3D structure of real scene， which causes potential risk to the long-duration running of SLAM system. We can resolve this contextual problem from the perspective of probability. The observation is considered as an invalid map point when the dynamic probability is lower than a certain threshold in terms of the motion probability of the previous 2D observation. Our system can be used to alter the invalid map points in time， and it can run stably in highly dynamic scene for a long time.
Result
2
Our method proposed is focused on both quantitative and qualitative aspects. Compared to such conventional monocular vSLAM system， our method has its priority in absolute trajectory error （ATE） on TUM-RGBD high dynamic sequences. This metric is illustrated its effectiveness for real time applications in terms of such AR-related visual localization. Due to the motion range of the dynamic object （person） in these sequences is vulnerable， and the scenes are relatively similar， the VICON motion capture system is used to record our own challenging dynamic dataset. In these scenes with complicated object and multiview motion， our method has absolutely advantage in ATE and the integrity of the tracking. Additionally， we compare the quality of mapping and augmented reality（AR） application in highly dynamic scenes qualitatively as well. To maintain the consistency of the real environment while mapping， the sparse map of our method can change synchronously in related to the dynamic object moving in the scene， which is beneficial for some map-based downstream applications like robot path planning. For AR application， virtual cube can be fixed in the 3D scene stably， and it is not drifted with the disturb of dynamic objects in the scene.
Conclusion
2
By integrating semantic information into dynamic scene and robust estimation algorithm， the proposed method carries out data association and motion state detection on the segmentation areas， represents the motion state of 2D observation in the form of probability， and eliminates the invalid map points in time， which can significantly improve the accuracy of camera pose estimation， the quality of mapping and the robustness of monocular vSLAM in highly dynamic environment.
视觉SLAM（vSLAM）语义分割动态场景鲁棒性估计概率预测
visual-based simultaneous localization and mapping （vSLAM）semantic segmentationdynamic environmentrobust estimationprobability prediction
Alcantarilla P F， Yebes J J， Almazán J and Bergasa L M. 2012. On combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments//Proceedings of 2012 IEEE International Conference on Robotics and Automation. Saint Paul， USA： IEEE： 1290-1297 ［DOI： 10.1109/ICRA.2012.6224690http://dx.doi.org/10.1109/ICRA.2012.6224690］
Bescos B， Fácil J M， Civera J and Neira J. 2018. DynaSLAM： tracking， mapping， and inpainting in dynamic scenes. IEEE Robotics and Automation Letters， 3（4）： 4076-4083 ［DOI： 10.1109/LRA.2018.2860039http://dx.doi.org/10.1109/LRA.2018.2860039］
Bowman S L， Atanasov N， Daniilidis K and Pappas G J. 2017. Probabilistic data association for semantic SLAM//Proceedings of 2017 IEEE International Conference on Robotics and Automation. Singapore， Singapore： IEEE： 1722-1729 ［DOI： 10.1109/ICRA.2017.7989203http://dx.doi.org/10.1109/ICRA.2017.7989203］
Brasch N， Bozic A， Lallemand J and Tombari F. 2018. Semantic monocular SLAM for highly dynamic environments//Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid， Spain： IEEE： 393-400 ［DOI： 10.1109/IROS.2018.8593828http://dx.doi.org/10.1109/IROS.2018.8593828］
Cui L Y and Ma C W. 2019. SOF-SLAM： a semantic visual SLAM for dynamic environments. IEEE Access， 7： 166528-166539 ［DOI： 10.1109/ACCESS.2019.2952161http://dx.doi.org/10.1109/ACCESS.2019.2952161］
Fischler M A and Bolles R C. 1981. Random sample consensus： a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM， 24（6）： 381-395 ［DOI： 10.1145/358669.358692http://dx.doi.org/10.1145/358669.358692］
Hartley R and Zisserman A. 2003. Multiple View Geometry in Computer Vision. 2nd ed. New York： Cambridge University Press
He K M， Gkioxari G， Dollár P and Girshick R. 2017. Mask R-CNN//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 2980-2988 ［DOI： 10.1109/ICCV.2017.322http://dx.doi.org/10.1109/ICCV.2017.322］
Kaneko M， Iwami K， Ogawa T， Yamasaki T and Aizawa K. 2018. Mask-SLAM： robust feature-based monocular SLAM by masking using semantic segmentation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City， USA： IEEE： 258-266 ［DOI： 10.1109/CVPRW.2018.00063http://dx.doi.org/10.1109/CVPRW.2018.00063］
Li Y W， Zhao H S， Qi X J， Wang L W， Li Z M， Sun J and Jia J Y. 2021. Fully convolutional networks for panoptic segmentation//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 214-223 ［DOI： 10.1109/CVPR46437.2021.00028http://dx.doi.org/10.1109/CVPR46437.2021.00028］
Lin T Y， Maire M， Belongie S， Hays J， Perona P， Ramanan D， Dollár P and Zitnick C L. 2014. Microsoft COCO： common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich， Switzerland： Springer： 740-755 ［DOI： 10.1007/978-3-319-10602-1_48http://dx.doi.org/10.1007/978-3-319-10602-1_48］
Liu Y B and Miura J. 2021. RDS-SLAM： real-time dynamic SLAM using semantic segmentation methods. IEEE Access， 9： 23772-23785 ［DOI： 10.1109/ACCESS.2021.3050617http://dx.doi.org/10.1109/ACCESS.2021.3050617］
Lowe D G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision， 60（2）： 91-110 ［DOI： 10.1023/B：VISI.0000029664.99615.94http://dx.doi.org/10.1023/B：VISI.0000029664.99615.94］
Lucas B D and Kanade T. 1981. An iterative image registration technique with an application to stereo vision//Proceedings of the 7th International Joint Conference on Artificial Intelligence. Vancouver， Canada： Morgan Kaufmann Publishers Inc.
Mur-Artal R and Tardós J D. 2017. ORB-SLAM2： an open-source SLAM system for monocular， stereo， and RGB-D cameras. IEEE Transactions on Robotics， 33（5）： 1255-1262 ［DOI： 10.1109/TRO.2017.2705103http://dx.doi.org/10.1109/TRO.2017.2705103］
Oberkampf D， DeMenthon D F and Davis L S. 1993. Iterative pose estimation using coplanar points//Proceedings of 1993 IEEE Conference on Computer Vision and Pattern Recognition. New York， USA： IEEE： 626-627 ［DOI： 10.1109/CVPR.1993.341055http://dx.doi.org/10.1109/CVPR.1993.341055］
Runz M， Buffier M and Agapito L. 2018. MaskFusion： real-time recognition， tracking and reconstruction of multiple moving objects//Proceedings of 2018 IEEE International Symposium on Mixed and Augmented Reality. Munich， Germany： IEEE： 10-20 ［DOI： 10.1109/ISMAR.2018.00024http://dx.doi.org/10.1109/ISMAR.2018.00024］
Saputra M R U， Markham A and Trigoni N. 2019. Visual SLAM and structure from motion in dynamic environments： a survey. ACM Computing Surveys， 51（2）： #37 ［DOI： 10.1145/3177853http://dx.doi.org/10.1145/3177853］
Scona R， Jaimez M， Petillot Y R， Fallon M and Cremers D. 2018. StaticFusion： background reconstruction for dense RGB-D SLAM in dynamic environments//Proceedings of 2018 IEEE International Conference on Robotics and Automation. Brisbane， Australia： IEEE： 3849-3856 ［DOI： 10.1109/ICRA.2018.8460681http://dx.doi.org/10.1109/ICRA.2018.8460681］
Sheng C， Pan S G， Zhao T， Zen P and Huang L X. 2020. Monocular SLAM system in dynamic scenes based on image semantic segmentation. Bulletin of Surveying and Mapping， （1）： 40-44
盛超， 潘树国， 赵涛， 曽攀， 黄砺枭. 2020. 基于图像语义分割的动态场景下的单目SLAM算法. 测绘通报， （1）： 40-44 ［DOI： 10.13474/j.cnki.11-2246.2020.0009http://dx.doi.org/10.13474/j.cnki.11-2246.2020.0009］
Sturm J， Engelhard N， Endres F， Burgard W and Cremers D. 2012. A benchmark for the evaluation of RGB-D SLAM systems//Proceedings of 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Vilamoura-Algarve， Portugal： IEEE： 573-580 ［DOI： 10.1109/IROS.2012.6385773http://dx.doi.org/10.1109/IROS.2012.6385773］
Tan W， Liu H M， Dong Z L， Zhang G F and Bao H J. 2013. Robust monocular SLAM in dynamic environments//Proceedings of 2013 IEEE International Symposium on Mixed and Augmented Reality. Adelaide， Australia： IEEE： 209-218 ［DOI： 10.1109/ISMAR.2013.6671781http://dx.doi.org/10.1109/ISMAR.2013.6671781］
Yu C， Liu Z X， Liu X J， Xie F G， Yang Y， Wei Q and Fei Q. 2018. DS-SLAM： a semantic visual SLAM towards dynamic environments//Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid， Spain： IEEE： 1168-1174 ［DOI： 10.1109/IROS.2018.8593691http://dx.doi.org/10.1109/IROS.2018.8593691］
Zhang S S， Zheng L Y and Tao W B. 2021. Survey and evaluation of RGB-D SLAM. IEEE Access， 9： 21367-21387 ［DOI： 10.1109/ACCESS.2021.3053188http://dx.doi.org/10.1109/ACCESS.2021.3053188］
Zou D P and Tan P. 2013. CoSLAM： collaborative visual SLAM in dynamic environments. IEEE Transactions on Pattern Analysis and Machine Intelligence， 35（2）： 354-366 ［DOI： 10.1109/TPAMI.2012.104http://dx.doi.org/10.1109/TPAMI.2012.104］
相关文章
相关作者
相关机构
京公网安备11010802024621