深度学习点云质量增强方法综述
陈建文,赵丽丽,任蓝草,孙卓群,张新峰,马思伟(电子科技大学信息与通信工程学院;中国移动通信集团有限公司研究院;中国科学院大学计算机科学与技术学院;北京大学计算机学院) 摘 要
随着三维探测技术的发展,点云逐渐成为最常用的三维物体/场景表征数据类型之一,被广泛地应用于自动驾驶、增强现实及虚拟现实等领域中。然而,受限于硬件设备、采集环境以及遮挡等因素,采集的原始点云通常是不完整、稀疏、嘈杂的,为点云的处理和分析带来了巨大挑战。在此背景下,点云质量增强技术旨在对原始点云进行处理以获得结构完整、密集且接近无噪的点云,具有重要意义。本文对现阶段基于深度学习的点云质量增强方法进行了系统综述,为后续研究者提供研究基础。首先,简要介绍了点云数据处理中通用的关键技术;分别介绍了补全、上采样和去噪三类点云质量增强方法,并对三类方法中的现有算法进行了分类、梳理及总结。其中,点云补全与点云去噪算法均可根据是否采用编码器-解码器结构分为两大类,点云上采样算法可根据网络主要结构分为基于卷积神经网络的方法、基于生成对抗网络的方法和基于图卷积神经网络的方法。其次,总结了质量增强任务中常用的数据集与评价指标,并分别对比分析了现阶段点云补全、上采样和去噪中主流算法的性能。最后,通过系统的梳理,凝练出点云质量增强方向所面临的挑战,对未来的研究点与趋势进行了展望。
关键词
Deep learning-based quality enhancement for 3D point clouds: A survey
Chen Jianwen,Zhao Lili,Ren Lancao,Sun Zhuoqun,Zhang Xinfeng,Ma Siwei(China Mobile Research Institute) Abstract
With the development of three dimensional (3D) detection technologies, point clouds have gradually become one of the most common data representations of 3D objects or scenes, which are widely used in many applications such as automatic drive, augmented reality (AR) and virtual reality (VR), etc. However, limited by the hardware equipment, environment and occlusion, the acquired point clouds are usually sparse, noisy and uneven, which imposes great challenges into the pro-cessing and analysis of point clouds. Therefore, point cloud quality enhancement techniques, which aim to process the original point cloud to obtain a dense, clean, and structurally-complete point cloud, are of great significance. In recent years, with the development of hardware equipment and machine learning technologies, the deep learning-based point cloud quality enhancement methods, which have the great potential to extract the features of point clouds, have attracted the at-tention of scholars at home and abroad. The related works mainly include point cloud completion, point cloud upsampling (also known as super-resolution) and point cloud denoising. Point cloud completion refers to the completion of incomplete point clouds to restore the complete point cloud information. Point cloud upsampling refers to increasing the point number of the original point cloud to get a denser one, while point cloud denoising refers to removing the noisy points existing in the point cloud to get a cleaner one. This paper systematically reviews the existing point cloud quality enhancement meth-ods based on deep learning, which provides a research basis for subsequent researchers. First, this study briefly introduces the fundamentals and key technologies that are widely used in point cloud analysis. Then, three types of point cloud qual-ity enhancement technologies, i.e., upsampling, completion and denoising, are introduced respectively, which are further classified and summarized. According to the type of input data, point cloud completion methods can be divided into the voxel-based algorithms and the point-based algorithms. The point-based algorithms can be further divided into two types based on whether the encoder-decoder structure is exploited or not. The encoder-decoder structure-based algorithms can be further divided according to whether the generative adversarial network (GAN) structure is used. Point cloud upsampling methods can be classified into the convolution neural network (CNN)-based algorithms, the GAN-based algorithms and the graph convolution network (GCN)-based algorithms. Point cloud denoising methods can also be divided into two types based on whether the encoder-decoder structure is exploited or not. Second, the commonly-used datasets and evaluation metrics in point cloud quality enhancement tasks are summarized. For the performance evaluation, the evaluation metrics for geometry reconstruction are illustrated in detail, which mainly consist of Chamfer Distance (CD), Earth Mover’s Dis-tance (EMD), Hausdorff Distance (HD) and Point to Surface Distance (P2F). On the common datasets, this paper compares the state-of-the-art algorithms of point cloud completion and upsampling, respectively, and the reasons for the different per-formance of algorithms are analyzed. At the end, the recent progress and challenges are summarized, and the future research trends are prospected. 1) The point cloud features extracted by existing deep learning-based algorithms are more global. This means that local features related to the detailed structure can not captured well, resulting in poor performance of detail reconstruction. It is well-known that traditional geometric algorithms can effectively represent data features based on geo-metric information. Therefore, how to combine geometric algorithms and deep learning for point cloud quality enhancement is one of the key problems worthy of exploration. 2) Most algorithms are for dense point clouds of single objects, and there is very few research on sparse LiDAR point clouds containing large-scale outdoor scenes. 3) Most works only consider point cloud processing of a single frame, and ignore the temporal correlation of point cloud sequences. Therefore, it is a worthwhile problem to explore how to utilize the spatial-temporal correlation to improve the performance of quality en-hancement. 4) In the existing methods, the proposed network models are often complex and the inference speed is relatively slow, which can not meet the real-time requirements of the applications. Therefore, how to further reduce the scale of the model parameters and improve the inference speed is one of the directions worthy of exploring. 5) Most of methods only process the geometric information (3D coordinates) of point clouds, ignoring the attribute information (color, normals, reflection intensity, etc.). Therefore, it will be of great significance to study how to simultaneously conduct quality en-hancement of geometric information and attribute information.
Keywords
|