浏览全部资源
扫码关注微信
Received：16 June 2020，
Revised：2020-8-13，
Accepted：20 August 2020，
Published：16 April 2021
移动端阅览
目的
2
3D点云与以规则的密集网格表示的图像不同，不仅不规则且无序，而且由于输入输出大小和顺序差异，具有密度不均匀以及形状和缩放比例存在差异的特性。为此，提出一种对3D点云进行卷积的方法，将关系形状卷积神经网络（relation-shape convolution neural network，RSCNN）与逆密度函数相结合，并在卷积网络中增添反卷积层，实现了点云更精确的分类分割效果。
方法
2
在关系形状卷积神经网络中，将卷积核视为由权重函数和逆密度函数组成的3D点局部坐标的非线性函数。对给定的点，权重函数通过多层感知器网络学习，逆密度函数通过核密度估计（kernel density estimation，KDE）学习，逆密度函数的引入对点云采样率不均匀的情况进行弥补。在点云分割任务中，引入由插值和关系形状卷积层两部分组成的反卷积层，将特征从子采样点云传播回原始分辨率。
结果
2
在ModelNet40、ShapeNet、ScanNet数据集上进行分类、部分分割和语义场景分割实验，验证模型的分类分割性能。在分类实验中，与PointNet++相比，整体精度提升3.1%，在PointNet++将法线也作为输入的情况下，精度依然提升了1.9%；在部分分割实验中，类平均交并比（mean intersection over union，mIoU）比PointNet++在法线作为输入情况下高6.0%，实例mIoU比PointNet++高1.4%；在语义场景分割实验中，mIoU比PointNet++高13.7%。在ScanNet数据集上进行不同步长有无逆密度函数的对比实验，实验证明逆密度函数将分割精度提升0.8%左右，有效提升了模型性能。
结论
2
融合逆密度函数的关系形状卷积神经网络可以有效获取点云数据中的局部和全局特征，并对点云采样不均匀的情况实现一定程度的补偿，实现更优的分类和分割效果。
Objective
2
3D point clouds have received widespread attention for their wide range of applications
such as robotics
autonomous driving
and virtual reality applications. However
due to the characteristics of input and output size and order differences
uneven density
and differences in shape and scaling
point cloud processing is very challenging
and because of this
various shapes formed by irregular points are often difficult to distinguish. For this problem
sufficient contextual semantic information must be captured to thoroughly grasp the elusive shapes. In 2D images
convolutional neural networks (CNN) have fundamentally changed the landscape of computer vision by greatly improving the results of almost all vision tasks. CNN succeeds by using translation invariance
so the same set of convolution filters can be applied to all positions in the image
thereby reducing the number of parameters and improving the generalization ability. We hope to transfer these successes to 3D point cloud analysis. However
a 3D point cloud is an unordered set of 3D points
each with or without additional features (such as RGB)
which does not conform to the regular lattice grid in a 2D image. Applying conventional CNNs to such unordered inputs is difficult. Some works convert point clouds into regular voxels or multi-view images. However
these conversions usually cause a large amount of inherent geometric information to be lost
and the amount of data in the 3D point cloud is huge
increasing the complexity of the conversion. Another solution is to learn directly from an irregular point cloud. PointNet learns each point independently
and then applies a symmetric function to accumulate features. This direction can realize the invariance of point replacement. Although impressive
it ignores local patterns that have proven to be important for advanced visual semantics in abstract image CNNs. To correct this
KCNet mines local patterns by creating KNN (K-nearest neighbor) graphs at each point of PointNet. However
it does not have a pooling layer that can explicitly increase the semantic level. PointNet++ hierarchical groups point clouds into local subsets and learn them through PointNet. This design does work similar to CNN
but the basic operation of PointNet requires high complexity to achieve sufficient effectiveness
which usually results in huge computational costs.
Method
2
To solve this problem
some works divide the point cloud into several subsets by sampling. Then
a hierarchical structure is constructed to learn the context representation from local to global. However
this greatly depends on effective inductive learning of local subsets
which is difficult to achieve for point clouds with uneven density and irregular shape. Inspired by inverse density function in point cloud networks
we propose a relational shape CNN (RSCNN) fused with inverse density function. The key of the network is to learn from the relationship
that is
to learn from the geometric topology constraints between the points
which encode meaningful shape information in the 3D point cloud
and the convolution kernel is regarded as a nonlinear function of the local coordinates of 3D points composed of 3D points. For a given point
predefined geometric prior convolution weights are used in advanced relational expressions
and kernel density estimation is used to study inverse density functions. The introduction of the inverse density function is further applied to non-uniformly sampled point clouds. In addition
for the segmentation task
we propose a deconvolution operator. Relation-shape deconvolution layer(RSDeconv) consists of two parts: interpolation and Relation-shape convolution layer(RSConv). The feature is propagated from the sampling point cloud to the original resolution through convolution. The spatial layout of the points can be used to calculate the induction through convolution
and it can reflect the potential shape formed by irregular points differently
so as to realize the context shape awareness learning of point cloud analysis. Benefit from the geometric prioris
the invariance to point displacement and the robustness to rigid transformations (such as translation and rotation) are realized.
Result
2
Classification
partial segmentation
and semantic scene segmentation experiments were conducted on ModelNet40
ShapeNet
and ScanNet datasets to verify the classification and segmentation performance of the model. In the classification experiment of ModelNet40
its overall accuracy is improved by 3.1% compared with PointNet++. Even when PointNet++ takes normal as input
its accuracy is improved by 1.9% compared with PointNet++. In the ShapeNet partially segmented dataset
the mean intersection-over-union (mIoU) is 6.0% higher than that of PointNet++
and the instance mIoU is 1.4% higher than that of PointNet++. In the ScanNet indoor scene dataset
the mIoU is 13.7% higher than that of PointNet++. A comparison experiment is conducted on the ScanNet dataset to compare the unsynchronized length with and without density function. The experiment proves that the inverse density function improves the segmentation accuracy by about 0.8% and effectively improves the model performance.
Conclusion
2
The experimental results show that the heterogeneous sampling of point clouds can be realized by introducing the inverse density function into the relational CNN. In addition
the deconvolution layer further propagates the feature from the subsampling point cloud back to the original resolution
which improves the segmentation accuracy. In general
the proposed network model can effectively obtain the global and local characteristics of point cloud data
achieving better classification and segmentation effect.
Atzmon M, Maron H and Lipman Y. 2018. Point convolutional neural networks by extension operators. ACM Transactions on Graphics, 37(4): #71
Chen L C, Papandreou G, Kokkinos I, Murphy K and Yuille A L. 2018. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834-848[DOI:10.1109/TPAMI.2017.2699184]
Dai A, Chang A X, Savva M, Halber M, Funkhouser T and Nieβner M. 2017. ScanNet: richly-annotated 3D reconstructions of indoor scenes//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 2432-2443[ DOI: 10.1109/CVPR.2017.261 http://dx.doi.org/10.1109/CVPR.2017.261 ]
Hermosilla P, Ritschel T, Vázquez P P, Vinacua À and Ropinski T. 2018. Monte Carlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Graphics, 37(6): #235[DOI:10.1145/3272127.3275110]
Hu H, Gu J Y, Zhang Z, Dai J F and Wei Y C. 2018. Relation networks for object detection//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 3588-3597[ DOI: 10.1109/CVPR.2018.00378 http://dx.doi.org/10.1109/CVPR.2018.00378 ]
Huang K Q, Ren W Q and Tan T N. 2014. A review on image object classification and detection. Chinese Journal of Computers, 37(6): 1225-1240
黄凯奇, 任伟强, 谭铁牛. 2014. 图像物体分类与检测算法综述. 计算机学报, 37(6): 1225-1240 [DOI:10.3724/SP.J.1016.2014.01225]
Jaderberg M, Simonyan K, Zisserman A and Kavukcuoglu K. 2015. Spatial transformer networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press: 2017-2025
Jampani V, Kiefel M and Gehler P V. 2016. Learning sparse high dimensional filters: image filtering, dense CRFs and bilateral neural networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 4452-4461[ DOI: 10.1109/CVPR.2016.482 http://dx.doi.org/10.1109/CVPR.2016.482 ]
Klokov R and Lempitsky V. 2017. Escape from cells: deep Kd-networks for the recognition of 3D point cloud models//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE: 863-872[ DOI: 10.1109/ICCV.2017.99 http://dx.doi.org/10.1109/ICCV.2017.99 ]
Li X, Wang M Y, Wen C C, Wang L J, Zhou N and Fang Y. 2019. Density-aware convolutional networks with context encoding for airborne LiDAR point cloud classification[EB/OL].[2020-08-10] . https://arxiv.org/pdf/1910.05909.pdf https://arxiv.org/pdf/1910.05909.pdf
Li Y Y, Bu R, Sun M C, Wu W, Di X H and Chen B Q. 2018. PointCNN: convolution on X -transformed points[EB/OL].[2020-08-13] . https://arxiv.org/pdf/1801.07791.pdf https://arxiv.org/pdf/1801.07791.pdf .
Liu Y C, Fan B, Xiang S M and Pan C H. 2019. Relation-shape convolutional neural network for point cloud analysis//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 8887-8896[ DOI: 10.1109/CVPR.2019.00910 http://dx.doi.org/10.1109/CVPR.2019.00910 ]
Maturana D and Scherer S. 2015. VoxNet: a 3D convolutional neural network for real-time object recognition//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany: IEEE: 922-928[ DOI: 10.1109/IROS.2015.7353481 http://dx.doi.org/10.1109/IROS.2015.7353481 ]
Noh H, Hong S and Han B. 2015. Learning deconvolution network for semantic segmentation//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE: 1520-1528[ DOI: 10.1109/ICCV.2015.178 http://dx.doi.org/10.1109/ICCV.2015.178 ]
Parzen E.1962. On estimation of a probability density function and mode. Annals of Mathematical Statistics, 33(3): 1065-1076[DOI:10.1214/aoms/1177704472]
Qi C R, Liu W, Wu C X, Su H and Guibas L J. 2018. Frustum PointNets for 3D object detection from RGB-D data//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 918-927[ DOI: 10.1109/CVPR.2018.00102 http://dx.doi.org/10.1109/CVPR.2018.00102 ]
Qi C R, Su H, Nieβner M, Dai A, Yan M Y and Guibas L J. 2016. Volumetric and multi-view CNNS for object classification on 3D data//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 5648-5656[ DOI: 10.1109/CVPR.2016.609 http://dx.doi.org/10.1109/CVPR.2016.609 ]
Qi C R, Su H, Kaichun M and Guibas L J. 2017a. PointNet: deep learning on point sets for 3D classification and segmentation//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 77-85[ DOI: 10.1109/CVPR.2017.16 http://dx.doi.org/10.1109/CVPR.2017.16 ]
Qi C R, Yi L, Su H and Guibas L J. 2017b. PointNet++: deep hierarchical feature learning on point sets in a metric space[EB/OL].[2020-07-25] . https://arxiv.org/pdf/1706.02413.pdf https://arxiv.org/pdf/1706.02413.pdf
Riegler G, Ulusoy A O and Geiger A. 2017. OctNet: learning deep 3D representations at high resolutions//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu,USA: IEEE: 6620-6629[ DOI: 10.1109/CVPR.2017.701 http://dx.doi.org/10.1109/CVPR.2017.701 ]
Shen Y R, Feng C, Yang Y Q and Tian D. 2018. Mining point cloud local structures by kernel correlation and graph pooling//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 4548-4557[ DOI: 10.1109/CVPR.2018.00478 http://dx.doi.org/10.1109/CVPR.2018.00478 ]
Su H, Jampani V, Sun D Q, Maji S, Kalogerakis E, Yang M H and Kautz J. 2018. SPLATNet: sparse lattice networks for point cloud processing//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 2530-2539[ DOI: 10.1109/CVPR.2018.00268 http://dx.doi.org/10.1109/CVPR.2018.00268 ]
Su H, Maji S, Kalogerakis E and Learned-Miller E. 2015. Multi-view convolutional neural networks for 3D shape recognition//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE: 945-953[ DOI: 10.1109/ICCV.2015.114 http://dx.doi.org/10.1109/ICCV.2015.114 ]
Wang X L, Girshick R, Gupta A and He K M. 2018. Non-local neural networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7794-7803[ DOI: 10.1109/CVPR.2018.00813 http://dx.doi.org/10.1109/CVPR.2018.00813 ]
Wang Y, Sun Y B, Liu Z W, Sarma S E, Bronstein M M and Solomon J M. 2019. Dynamic graph CNN for learning on point clouds. ACM Transactions on Graphics, 38(5): #146
Wen W W, Wen G J, Hui B W and Chen D X. 2019. Model library construction by combining global and local surfaces for 3D object recognition. Journal of Image and Graphics, 24(2): 248-257
文威威, 文贡坚, 回丙伟, 陈鼎新. 2019. 结合全局与局部信息的点云目标识别模型库构建. 中国图象图形学报, 24(2): 248-257 [DOI:10.11834/jig.180270]
Wu W X, Qi Z G and Li F X. 2019. PointConv: deep convolutional networks on 3D point clouds//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 9613-9622[ DOI: 10.1109/CVPR.2019.00985 http://dx.doi.org/10.1109/CVPR.2019.00985 ]
Wu Z R, Song S R, Khosla A, Yu F, Zhang L G, Tang X O and Xiao J X. 2015. 3D ShapeNets: a deep representation for volumetric shapes//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE: 1912-1920[ DOI: 10.1109/CVPR.2015.7298801 http://dx.doi.org/10.1109/CVPR.2015.7298801 ]
Xu Y F, FanT Q, Xu M Y, Zeng L and Qiao Y. 2018. SpiderCNN: deep learning on point sets with parameterized convolutional filters[EB/OL].[2020-08-06] . https://arxiv.org/pdf/1803.11527.pdf https://arxiv.org/pdf/1803.11527.pdf
Yi L, Kim V G, Ceylan D, Shen I C, Yan M Y, Su H, Lu C W, Huang Q X, Sheffer A and Guibas L. 2016. A scalable active framework for region annotation in 3D shape collections. ACM Transactions on Graphics, 35(6): #210[DOI:10.1145/2980179.2980238]
Yuan X C, Chen H W and Li Y W. 2017. Normal estimation method for regular point cloud surface with sharp feature. Journal of Image and Graphics, 22(3): 334-341
袁小翠, 陈华伟, 李彧雯. 2017. 规则特征曲面点云法向估计. 中国图象图形学报, 22(3): 334-341 [DOI:10.11834/jig.20170307]
Zhu Y K, Mottaghi R, Kolve E, Lim J J, Gupta A, Li F F and Farhadi A. 2017. Target-driven visual navigation in indoor scenes using deep reinforcement learning//Proceedings of 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, Singapore: IEEE: 3357-3364[ DOI: 10.1109/ICRA.2017.7989381 http://dx.doi.org/10.1109/ICRA.2017.7989381 ]
相关文章
相关作者
相关机构
京公网安备11010802024621
