浏览全部资源
扫码关注微信
收稿：2020-08-20，
修回：2020-9-18，
录用：2020-9-25，
纸质出版：2021-12-16
移动端阅览
目的
2
在点云场景中，语义分割对场景理解来说是至关重要的视觉任务。由于图像是结构化的，而点云是非结构化的，点云上的卷积通常比图像上的卷积更加困难，会消耗更多的计算和内存资源。在这种情况下，大尺度场景的分割往往需要分块进行，导致效率不足并且无法捕捉足够的场景信息。为了解决这个问题，本文设计了一种计算高效且内存高效的网络结构，可以用于端到端的大尺度场景语义分割。
方法
2
结合空间深度卷积和残差结构设计空间深度残差（spatial depthwise residual，SDR）块，其具有高效的计算效率和内存效率，并且可以有效地从点云中学习到几何特征。另外，设计一种扩张特征整合（dilated feature aggregation，DFA）模块，可以有效地增加感受野而仅增加少量的计算量。结合SDR块和DFA模块，本文构建SDRNet（spatial depthwise residual network），这是一种encoder-decoder深度网络结构，可以用于大尺度点云场景语义分割。同时，针对空间卷积核输入数据的分布不利于训练问题，提出层级标准化来减小参数学习的难度。特别地，针对稀疏雷达点云的旋转不变性，提出一种特殊的SDR块，可以消除雷达数据绕
Z
轴旋转的影响，显著提高网络处理激光雷达点云时的性能。
结果
2
在S3DIS（stanford large-scale 3D indoor space）和SemanticKITTI（Karlsruhe Institute of Technology and Toyota Technological Institute）数据集上对提出的方法进行测试，并分析点数与帧率的关系。本文方法在S3DIS数据集上的平均交并比（mean intersection over union，mIoU）为71.7%，在SemanticKITTI上的mIoU在线单次扫描评估中达到59.1%。
结论
2
实验结果表明，本文提出的SDRNet能够直接在大尺度场景下进行语义分割。在S3DIS和SemanticKITTI数据集上的实验结果证明本文方法在精度上有较好表现。通过分析点数量与帧率之间的关系，得到的数据表明本文提出的SDRNet能保持较高精度和较快的推理速率。
Objective
2
Point cloud semantic segmentation has been an essential visual task for scene understanding from two-dimensional vision to three-dimensional vision. Deep learning processing point cloud has been divided into three methods as following: point-based method
projection-based method and voxel-based method. Projection-based methods have obtained a two-dimensional image from the point cloud based on spherical projection. The semantic segmentation on the point cloud has been conducted via a two-dimensional convolution neural network method. The original point cloud has been restored via some post-processing. However
those methods have usually only been used for LiDAR point clouds. Voxel-based methods have often consumed a lot of memory due to voxel representation. The above two methods have both represented the unstructured point cloud into a structured form and processed it via a two-dimensional convolutional neural network or a three-dimensional convolutional neural network. However
this method will lose geometric details. Point-based methods have often consumed more memory subjected to additional neighborhood information storage. Some existing methods have usually divided the entire point cloud into blocks for processing. However
this method will destroy the geometric structure of the scene to cause incomplete information capture from the scene. In addition
some point-based methods in large-scale scenes have the problem of insufficient receptive fields caused by shallow network structures due to excessive memory consumption. A computation-based and memory-efficient network structure has been presented that can be used for end-to-end large-scale scene semantic segmentation.
Method
2
The spatial depthwise residual (SDR) block has been designed via combining the spatial depthwise convolution and residual structure to learn geometric features from the point cloud effectively. The receptive field has been regarded as one of the key factors in semantic segmentation. In order to increase the receptive field
a dilated feature aggregation (DFA) module
which has a larger receptive field than the SDR block
but with less calculation. The core idea of this module has reduced computational consumption and memory consumption via down sampling. Combining SDR block and DFA module
SDRNet
a deeper encoder-decoder network structure has been constructed
which can be applied to large-scale scenes semantic segmentation. The data distribution of the input data has affected the training process of the network. Data distribution is not conducive to network learning based on the analysis of input data of the convolution kernel. Hierarchical normalization (HN) can reduce the learning difficulty of the convolution kernel. A special SDR block has been used for a kind of rotation invariance of sparse LiDAR point clouds. Before convolution
the point and its neighborhood have been first rotated to a fixed angle. The influence of the rotation of the radar data around the
Z
-axis can be eliminated. The prediction result has not be changed via the rotated point cloud around the
Z
-axis. This special SDR block can significantly improve the performance of the network when processing LiDAR point clouds.
Result
2
The stanford large-scale 3D indoor space(S3DIS) dataset and the Karlsruhe Institute of Technology and Toyota Technological Institute(SemanticKITTI) dataset have been used. Different parameters for different tasks to adapt to the application scenarios of the task have been setup. A larger model for higher accuracy has been constructed because the S3DIS task has been focused on accuracy. The SemanticKITTI scene has required more speed. A lighter hyperparameter has been chosen. The designed model has been compared with several state-of-the-art models on the S3DIS datasets by using 6-flod cross validation. Mean intersection over union (mIoU)
mean accuracy (mAcc) and overall accuracy (OA) have been evaluated on the S3DIS dataset. The method has achieved 88.9% OA
82.4% mAcc and 71.7% mIoU each. These methods have presented well on different metrics. The online single scan evaluation has been conducted on the SemanticKITTI dataset. 59.1% mIoU has been obtained. The method has achieved better results in mIoU and several accuracy of several classes in comparison with point-based methods and projection-based methods. In an unmanned driving scenario like SemanticKITTI
the inference speed of the mode is a crucial factor. In addition
the inference speed of SDRNet with the different number of points has been tested. When the number of points is 50 K
the network processing point cloud speed can reach 11 frames per second (fps) by using a machine with NVIDIA RTX and i7-8700K. Moreover
this paper has constructed experiment of ablation study to explain the performance of each part of the model further.
Conclusion
2
The experiments on the S3DIS dataset SemanticKITTI dataset have shown that the research method can directly perform semantic segmentation in large-scale point cloud scenes. It can extract information from the scene effectively and achieve high accuracy. The experiment of ablation study on S3DIS area-5 has demonstrated that both DFA and HN can improve performance. The experiment of ablation study on SemanticKITTI validation set has presented that eliminating the influence of rotation by using the special SDR block can effectively improve the performance of the network. The higher accuracy and a relatively fast speed have been achieved via analyzing the relationship between the number of points and the frame rate.
Alonso I, Riazuelo L, Montesano L and Murillo A C. 2020a. 3D-mininet: learning a 2D representation from point clouds for fast and efficient 3D LIDAR semantic segmentation[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/2002.10893.pdf https://arxiv.org/pdf/2002.10893.pdf
Alonso I, Riazuelo L and Murillo A C. 2020b. MiniNet: an efficient semantic segmentation ConvNet for real-time robotic applications. IEEE Transactions on Robotics, 36(4): 1340-1347[DOI: 10.1109/TRO.2020.2974099]
Armeni I, Sax S, Zamir A R and Savarese S. 2017. Joint 2D-3D-semantic data for indoor scene understanding[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1702.01105.pdf https://arxiv.org/pdf/1702.01105.pdf
Behley J, Garbade M, Milioto A, Quenzel J, Behnke S, Stachniss C and Gall J. 2019. SemanticKITTI: a dataset for semantic scene understanding ofLiDAR sequences//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 9297-9306[ DOI: 10.1109/ICCV.2019.00939 http://dx.doi.org/10.1109/ICCV.2019.00939 ]
Bentley J L. 1975. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9): 509-517[DOI: 10.1145/361002.361007]
Boulch A, Puy G and Marlet R. 2020. FKAConv: feature-kernel alignment for point cloud convolution[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/2004.04462.pdf https://arxiv.org/pdf/2004.04462.pdf
Engelmann F, Kontogianni T and Leibe B. 2020. Dilated point convolutions: on the receptive field size of point convolutions on 3D point clouds[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1907.12046.pdf https://arxiv.org/pdf/1907.12046.pdf
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Howard A G, Zhu M L, Chen B, Kalenichenko D, Wang W J, Weyand T, Andreetto M and Adam H. 2017. MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1704.04861.pdf https://arxiv.org/pdf/1704.04861.pdf
Hu Q Y, Yang B, Xie L H, Rosa S, Guo Y L, Wang Z H, Trigoni N and Markham A. 2020. RandLA-Net: efficient semantic segmentation of large-scale point clouds//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 11105-11114[ DOI: 10.1109/CVPR42600.2020.01112 http://dx.doi.org/10.1109/CVPR42600.2020.01112 ]
Iandola F N, Han S, Moskewicz M W, Ashraf K, Dally W J and Keutzer K. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 mb model size[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1602.07360.pdf https://arxiv.org/pdf/1602.07360.pdf
Kingma D P and Ba J. 2015. Adam: a method for stochastic optimization[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1412.6980v5.pdf https://arxiv.org/pdf/1412.6980v5.pdf
Krizhevsky A, Sutskever I and Hinton G E. 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84-90[DOI: 10.1145/3065386]
Landrieu L and Simonovsky M. 2018. Large-scale point cloud semantic segmentation with superpoint graphs//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 4558-4567[ DOI: 10.1109/CVPR.2018.00479 http://dx.doi.org/10.1109/CVPR.2018.00479 ]
Li Y Y, Bu R, Sun M C, Wu W, Di X H and Chen B Q. 2018. PointCNN: convolution on $$X$$ -transformed points//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal, Canada: Curran Associates Inc. : 828-838
Liang Z D, Yang M, Deng L Y, Wang C X and Wang B. 2019. Hierarchical depthwise graph convolutional neural network for 3D semantic segmentation of point clouds//Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Montreal, Canada: IEEE: 8152-8158[ DOI: 10.1109/ICRA.2019.8794052 http://dx.doi.org/10.1109/ICRA.2019.8794052 ]
Liu Z J, Tang H T, Lin Y J and Han S. 2019. Point-voxel CNN for efficient 3D deep learning//Proceedings of the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc. : 965-975
Long J, Shelhamer E and Darrell T. 2015. Fully convolutional networks for semantic segmentation//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 3431-3440[ DOI: 10.1109/CVPR.2015.7298965 http://dx.doi.org/10.1109/CVPR.2015.7298965 ]
Milioto A, Vizzo I, Behley J and Stachniss C. 2019. RangeNet++: fast and accurate LiDAR semantic segmentation//Proceedings of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Macau, China: IEEE: 4213-4220[ DOI: 10.1109/IROS40897.2019.8967762 http://dx.doi.org/10.1109/IROS40897.2019.8967762 ]
Qi C R, Su H, Mo K and Guibas L J. 2017a. PointNet: deep learning on point sets for 3D classification and segmentation//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 77-85[ DOI: 10.1109/CVPR.2017.16 http://dx.doi.org/10.1109/CVPR.2017.16 ]
Qi C R, Yi L, Su H and Guibas L J. 2017b. PointNet++: deep hierarchical feature learning on point sets in a metric space//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, USA: Curran Associates Inc. : 5105-5114
Redmon J and Farhadi A. 2018. YOLOv3: an incremental improvement[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1804.02767.pdf https://arxiv.org/pdf/1804.02767.pdf
Rethage D, Wald J, Sturm J, Navab N and Tombari F. 2018. Fully-convolutional point networks for large-scale point clouds//Proceedings of 2018 European Conference on Computer Vision. Munich, Germany: Springer: 625-640[ DOI: 10.1007/978-3-030-01225-0_37 http://dx.doi.org/10.1007/978-3-030-01225-0_37 ]
Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241[ DOI: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf
Su H, Jampani V, Sun D Q, Maji S, Kalogerakis E, Yang M H and Kautz J. 2018. SPLATNet: sparse lattice networks for point cloud processing//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2530-2539[ DOI: 10.1109/CVPR.2018.00268 http://dx.doi.org/10.1109/CVPR.2018.00268 ]
Tchapmi L, Choy C, Armeni I, Gwak J and Savarese S. 2017. SEGCloud: semantic segmentation of 3D point clouds//Proceedings of 2017 International Conference on 3D Vision (3DV). Qingdao, China: IEEE: 537-547[ DOI: 10.1109/3DV.2017.00067 http://dx.doi.org/10.1109/3DV.2017.00067 ]
Thomas H, Qi CR, Deschaud J E, Marcotegui B, Goulette F and Guibas L. 2019. KPConv: flexible and deformable convolution for point clouds//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 6410-6419[ DOI: 10.1109/ICCV.2019.00651 http://dx.doi.org/10.1109/ICCV.2019.00651 ]
Wang Y, Sun Y B, Liu Z W, Sarma S E, Bronstein M M and Solomon J M. 2019. Dynamic graph CNN for learning on point clouds. ACM Transactions on Graphics, 38(5): #146[DOI: 10.1145/3326362]
Wu B C, Wan A, Yue X Y and Keutzer K. 2018. SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud//Proceedings of 2018 IEEE International Conference on Robotics and Automation (ICRA). Brisbane, Australia: IEEE: 1887-1893[ DOI: 10.1109/ICRA.2018.8462926 http://dx.doi.org/10.1109/ICRA.2018.8462926 ]
Wu B C, Zhou X Y, Zhao S C, Yue X Y and Keutzer K. 2019a. SqueezeSegV2: improved model structure and unsupervised domain adaptation for road-object segmentation from a LiDAR point cloud//Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Montreal, Canada: IEEE: 4376-4382[ DOI: 10.1109/ICRA.2019.8793495 http://dx.doi.org/10.1109/ICRA.2019.8793495 ]
Wu W X, Qi Z G and Fuxin L. 2019b. PointConv: deep convolutional networks on 3D point clouds//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 9613-9622[ DOI: 10.1109/CVPR.2019.00985 http://dx.doi.org/10.1109/CVPR.2019.00985 ]
Xu C F, Wu B C, Wang Z N, Zhan W, Vajda P, Keutzer K and Tomizuka M. 2020. SqueezeSegV3: spatially-adaptive convolution for efficient point-cloud segmentation[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/2004.01803.pdf https://arxiv.org/pdf/2004.01803.pdf
Yan X, Zheng C D, Li Z, Wang S and Cui S G. 2020. PointASNL: robust point clouds processing using nonlocal neural networks with adaptive sampling//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 5588-5597[ DOI: 10.1109/CVPR42600.2020.00563 http://dx.doi.org/10.1109/CVPR42600.2020.00563 ]
Yu C Q, Gao C X, Wang J B, Yu G, Shen C H and Sang N. 2020. BiseNet V2: bilateral network with guided aggregation for real-time semantic segmentation[EB/OL]. [2020-08-12] . https://arxiv.org/pdf/2004.02147.pdf https://arxiv.org/pdf/2004.02147.pdf
Zhang Y, Zhou Z X, David P, Yue X Y, Xi Z R, Gong B Q and Foroosh H. 2020. PolarNet: an improved grid representation for online LiDAR point clouds semantic segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 9598-9607[ DOI: 10.1109/CVPR42600.2020.00962 http://dx.doi.org/10.1109/CVPR42600.2020.00962 ]
Zhang Z Y, Hua B S and Yeung S K. 2019. ShellNet: efficient point cloud convolutional neural networks using concentric shells statistics//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 1607-1616[ DOI: 10.1109/ICCV.2019.00169 http://dx.doi.org/10.1109/ICCV.2019.00169 ]
相关作者
相关机构
京公网安备11010802024621
