Point cloud data classification and segmentation model using graph CNN and different pooling functions

Xinliang Zhang; Pengfei Fu; Yunji Zhao; Heng Xie; Wanru Wang

doi:10.11834/jig.190367

Image Analysis and Recognition | Views : 0 下载量: 22 CSCD: 4

PDF
Export
Share
Collection
Album

Point cloud data classification and segmentation model using graph CNN and different pooling functions
Vol. 25, Issue 6, Pages: 1201-1208(2020)
Received：19 July 2019，

Revised：2019-10-24，

Accepted：31 October 2019，

Published：16 June 2020
DOI： 10.11834/jig.190367
稿件说明：

移动端阅览

DOI：

Xinliang Zhang, Pengfei Fu, Yunji Zhao, Heng Xie, Wanru Wang. Point cloud data classification and segmentation model using graph CNN and different pooling functions[J]. Journal of Image and Graphics, 2020, 25(6): 1201-1208. DOI： 10.11834/jig.190367.

摘要

目的

深度网络用于3维点云数据的分类分割任务时，精度与模型在全局和局部特征上的描述能力密切相关。现有的特征提取网络，往往将全局特征和不同尺度下的局部特征相结合，忽略了点与点之间的结构信息和位置关系。为此，通过在分类分割模型中引入图卷积神经网络(graph convolution neural network，GCN)和改进池化层函数，增强局部特征表征能力和获取更丰富的全局特征，改善模型对点云数据的分类分割性能。

方法

GCN模块通过K近邻算法构造图结构，利用相邻点对的边缘卷积获取局部特征，在深度网络模型中动态扩展GCN使模型获得完备的局部特征。在池化层，通过选择差异性的池化函数，联合提取多个全局特征并进行综合，保证模型在数据抖动时的鲁棒性。

结果

在ModelNet40、ShapeNet和S3DIS(stanford large-scale 3D indoor semantics)数据集上进行分类、部分分割以及语义场景分割实验，验证模型的分类分割性能。与PointNet相比，在ModelNet40分类实验中，整体精度和平均分类精度分别提升4%和3.7%；在ShapeNet部分分割数据集和S3DIS室内场景数据集中，平均交并比(mean intersection-over-union，mIoU)分别高1.4%和9.8%。采用不同的池化函数测试结果表明，本文提出的差异性池化函数与PointNet提出的池化函数相比，平均分类精度提升了0.9%，有效改善了模型性能。

结论

本文改进的网络模型可以有效获取点云数据中的全局和局部特征，实现更优的分类和分割效果。

Abstract

Objective

The depth feature representation of the 3D model is the key and premise of 3D target recognition and 3D model semantic segmentation. It has broad application prospects in the fields of robot

automatic driving

virtual reality

and remote sensing mapping. Semantic segmentation has achieved great progress with the help of deep learning

but most of the methods are used to process 2D images. Given the large amount of data

uneven density

and irregular shape of unstructured 3D point clouds

their classification and segmentation still have enormous challenges. Traditional convolutional neural networks (CNNs) require regularized data as input. The point cloud needs to be converted into multi-view or a voxel mesh to process. The existing deep learning network used for directly processing point cloud data solves the disorder problem of point cloud through the pooling layer of CNN. Thus

the network model can directly classify and segment the point cloud data. As for the classification and segmentation model dealing with point cloud data

its accuracy is closely related to the ability of the network to describe global and local features. Existing feature extraction networks often combine global features with local features at different scales

ignoring the structural information and position relationship among points. Thus

the global feature vectors with more significant features cannot be generated in the pooling layer

resulting in low classification and segmentation accuracy.

Method

To improve the performance of the network model

the graph CNN (GCN) and the improved pooling layer function are introduced in the classification and segmentation model. The method can enhance the ability of local feature representation and obtain more abundant global features. The processing ability of the network model to point cloud data can be improved. In the GCN

a graph structure is constructed by connecting the vertex with the nearest

points through the K-nearest neighbor algorithm. The convolution operation is then carried out on the edge and relative position relationship of the adjacent point pairs in the graph structure. Consequently

the more detailed local features implicit in the point cloud data are extracted. The graph structure in the GCN model is not fixed. It is dynamically updated and the graph convolution module can be stacked numerous times in the network to further perceive the local characteristics of point cloud data. In the network pooling layer

a hybrid pooling structure is adopted composed of two parallel pooling channels to obtain the global feature vectors. The maximum pooling channel is used to obtain the maximal feature vector

while another maximum-average pooling channel is used to obtain a synthetic feature concerning the maximal and mean feature vectors. The acquired characteristic vectors are concatenated to obtain the final global feature vector of the network. Consequently

the network provides good robustness for the jittered data.

Result

The datasets ModelNet40

ShapeNet

and Stanford 3D indoor semantics (S3DIS) are mostly used for testing the performance of classification

partial segmentation

and semantic scene segmentation. Several experiments are carried out on the above three datasets to validate the performance of the model. In the classification experiment of ModelNet40

the proposed model achieves a better classification effect compared with the other competitive models. The overall accuracy and average classification accuracy are improved by 4% and 3.7%

respectively

compared with PointNet. In the partial segmentation ShapeNet dataset

the mean intersection-over-union (mIoU) is used as the index for evaluating model segmentation performance. In the comparison test

the proposed model in this study also obtains a satisfactory segmentation result. Specifically

our model's mIoU is 1.4% higher than that of PointNet. In S3DIS indoor scene dataset

our model's mIoU is 9.8% higher than PointNet. Furthermore

different pooling functions are tested and investigated to verify the effectiveness of the proposed hybrid pooling function in this study. Results show that the proposed hybrid pooling function in this study improves the average classification accuracy

exhibiting a 0.9% increase compared with the pooling function by PointNet.

Conclusion

Experimental results show that the local features of point cloud data can be effectively extracted by introducing GCN into the network model. The hybrid pooling function also yields great improvement in generating global characteristics with additional information. In general

the proposed network model can effectively obtain the global and local features of point cloud data and achieve better classification and segmentation effects.

关键词

Keywords

references

Armeni I, Sener O, Zamir A R, Jiang H, Brilakis I, Fischer M and Savarese S. 2016. 3D semantic parsing of large-scale indoor spaces//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 1534-1543[ DOI: 10.1109/CVPR.2016.170 http://dx.doi.org/10.1109/CVPR.2016.170 ]

Bai J, Si Q L and Qin F W. 2019. Lightweight real-time point cloud classification network LightPointNet. Journal of Computer-aided Design and Graphics, 31(4):612-621

白静, 司庆龙, 秦飞巍. 2019.轻量级实时点云分类网络LightPointNet.计算机辅助设计与图形学学报, 31(4):612-621)[DOI:10.3724/SP.J.1089.2019.17328]

Chen L C, Papandreou G, Kokkinos I, Murphy K and Yuille A L. 2018. DeepLab:semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-848[DOI:10.1109/TPAMI.2017.2699184]

Jaderberg M, Simonyan K, Zisserman A and Kavukcuoglu K. 2015. Spatial transformer networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press: 2017-2025

Jiang M Y, Wu Y R, Zhao T Q, Zhao Z L and Lu C W. 2018. PointSIFT: a SIFT-like network module for 3D point cloud semantic segmentation[EB/OL]. https://arxiv.org/pdf/1807.00652.pdf https://arxiv.org/pdf/1807.00652.pdf

Klokov R and Lempitsky V. 2017. Escape from cells: deep Kd-networks for the recognition of 3D point cloud models//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 863-872[ DOI: 10.1109/ICCV.2017.99 http://dx.doi.org/10.1109/ICCV.2017.99 ]

Li Y Y, Bu R, Sun M C, Wu W, Di X H and Chen B Q. 2018. PointCNN: convolution On X -transformed points[EB/OL]. https://arxiv.org/pdf/1801.07791.pdf https://arxiv.org/pdf/1801.07791.pdf .

Maturana D and Scherer S. 2015. VoxNet: a 3D convolutional neural network for real-time object recognition//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems. Hamburg, Germany: IEEE: 922-928[ DOI: 10.1109/IROS.2015.7353481 http://dx.doi.org/10.1109/IROS.2015.7353481 ]

Qi C R, Su H, Kaichun M and Guibas L J. 2017a. PointNet: deep learning on point sets for 3D classification and segmentation//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 77-85[ DOI: 10.1109/CVPR.2017.16 http://dx.doi.org/10.1109/CVPR.2017.16 ]

Qi C R, Su H, Nieβner M, Dai A, Yan M Y and Guibas L J. 2016. Volumetric and multi-view CNNS for object classification on 3D data//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 5648-5656[ DOI: 10.1109/CVPR.2016.609 http://dx.doi.org/10.1109/CVPR.2016.609 ]

Qi C R,Yi L, Su H and Guibas L J. 2017b. PointNet++: deep hierarchical feature learning on point sets in a metric space[EB/OL]. https://arxiv.org/pdf/1706.02413.pdf https://arxiv.org/pdf/1706.02413.pdf

Ren M W, Niu L and Fang Y. 2017. 3D-A-Nets: 3D deep dense descriptor for volumetric shapes with adversarial networks[EB/OL]. https://arxiv.org/pdf/1711.10108.pdf https://arxiv.org/pdf/1711.10108.pdf

Su H, Maji S, Kalogerakis E and Learned-Miller E. 2015. Multi-view convolutional neural networks for 3D shape recognition//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 945-953[ DOI: 10.1109/ICCV.2015.114 http://dx.doi.org/10.1109/ICCV.2015.114 ]

Wang R, Xue X Y, Ping X J, Niu S Z and Zhang T. 2018. Steganalysis of JPEG images based on image classification and segmentation. Journal ofImage and Graphics, 23(10):1472-1482

汪然, 薛小燕, 平西建, 牛少彰, 张涛. 2018.分类与分割相结合的JPEG图像隐写分析.中国图象图形学报, 23(10):1472-1482)[DOI:10.11834/jig.180037]

Wang Y, Sun Y B, Liu Z W, Sarma S E, Bronstein M M and Solomon J M. 2018. Dynamic graph CNN for learning on point clouds[EB/OL]. https: //arxiv.org/pdf/1801.07829.pdf

Wen W W, Wen G J, Hui B W and Chen D X. 2019. Model library construction by combining global and local surfaces for 3D object recognition. Journal of Image and Graphics, 24(2):248-257

文威威, 文贡坚, 回丙伟, 陈鼎新. 2019.结合全局与局部信息的点云目标识别模型库构建.中国图象图形学报, 24(2):248-257)[DOI:10.11834/jig.180270]

Wu Z R, Song S R, Khosla A, Yu F, Zhang L G, Tang X O and Xiao J X. 2015. 3D ShapeNets: a deep representation for volumetric shapes//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE: 1912-1920[ DOI: 10.1109/CVPR.2015.7298801 http://dx.doi.org/10.1109/CVPR.2015.7298801 ]

Yi L, Kim V G, Ceylan D, Shen I C, Yan M Y, Su H, Lu C W, Huang Q X, Sheffer A and Guibas L. 2016. A scalable active framework for region annotation in 3D shape collections. ACM Transactions on Graphics, 35(6):210[DOI:10.1145/2980179.2980238]

Yu B, Yin H T and Zhu Z X. 2017. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting//Proceedings of the 27th International Joint Conference on Artificial Intelligence.[s.l.]: IJCAI: 3634-3640[ DOI: 10.24963/ijcai.2018/505 http://dx.doi.org/10.24963/ijcai.2018/505 ]

Alert me when the article has been cited

提交

Enhanced attention-based joint semantic instance segmentation network for point clouds

Double-view feature fusion network for LiDAR semantic segmentation

Review on 3D point cloud registration method

A deep residual network with spatial depthwise convolution for large-scale point cloud semantic segmentation

Recent progress in rotation-invariant point cloud networks