Deep learning-based quality enhancement for 3D point clouds: A survey
Chen Jianwen,Zhao Lili,Ren Lancao,Sun Zhuoqun,Zhang Xinfeng,Ma Siwei(China Mobile Research Institute)
With the development of three dimensional (3D) detection technologies, point clouds have gradually become one of the most common data representations of 3D objects or scenes, which are widely used in many applications such as automatic drive, augmented reality (AR) and virtual reality (VR), etc. However, limited by the hardware equipment, environment and occlusion, the acquired point clouds are usually sparse, noisy and uneven, which imposes great challenges into the pro-cessing and analysis of point clouds. Therefore, point cloud quality enhancement techniques, which aim to process the original point cloud to obtain a dense, clean, and structurally-complete point cloud, are of great significance. In recent years, with the development of hardware equipment and machine learning technologies, the deep learning-based point cloud quality enhancement methods, which have the great potential to extract the features of point clouds, have attracted the at-tention of scholars at home and abroad. The related works mainly include point cloud completion, point cloud upsampling (also known as super-resolution) and point cloud denoising. Point cloud completion refers to the completion of incomplete point clouds to restore the complete point cloud information. Point cloud upsampling refers to increasing the point number of the original point cloud to get a denser one, while point cloud denoising refers to removing the noisy points existing in the point cloud to get a cleaner one. This paper systematically reviews the existing point cloud quality enhancement meth-ods based on deep learning, which provides a research basis for subsequent researchers. First, this study briefly introduces the fundamentals and key technologies that are widely used in point cloud analysis. Then, three types of point cloud qual-ity enhancement technologies, i.e., upsampling, completion and denoising, are introduced respectively, which are further classified and summarized. According to the type of input data, point cloud completion methods can be divided into the voxel-based algorithms and the point-based algorithms. The point-based algorithms can be further divided into two types based on whether the encoder-decoder structure is exploited or not. The encoder-decoder structure-based algorithms can be further divided according to whether the generative adversarial network (GAN) structure is used. Point cloud upsampling methods can be classified into the convolution neural network (CNN)-based algorithms, the GAN-based algorithms and the graph convolution network (GCN)-based algorithms. Point cloud denoising methods can also be divided into two types based on whether the encoder-decoder structure is exploited or not. Second, the commonly-used datasets and evaluation metrics in point cloud quality enhancement tasks are summarized. For the performance evaluation, the evaluation metrics for geometry reconstruction are illustrated in detail, which mainly consist of Chamfer Distance (CD), Earth Mover’s Dis-tance (EMD), Hausdorff Distance (HD) and Point to Surface Distance (P2F). On the common datasets, this paper compares the state-of-the-art algorithms of point cloud completion and upsampling, respectively, and the reasons for the different per-formance of algorithms are analyzed. At the end, the recent progress and challenges are summarized, and the future research trends are prospected. 1) The point cloud features extracted by existing deep learning-based algorithms are more global. This means that local features related to the detailed structure can not captured well, resulting in poor performance of detail reconstruction. It is well-known that traditional geometric algorithms can effectively represent data features based on geo-metric information. Therefore, how to combine geometric algorithms and deep learning for point cloud quality enhancement is one of the key problems worthy of exploration. 2) Most algorithms are for dense point clouds of single objects, and there is very few research on sparse LiDAR point clouds containing large-scale outdoor scenes. 3) Most works only consider point cloud processing of a single frame, and ignore the temporal correlation of point cloud sequences. Therefore, it is a worthwhile problem to explore how to utilize the spatial-temporal correlation to improve the performance of quality en-hancement. 4) In the existing methods, the proposed network models are often complex and the inference speed is relatively slow, which can not meet the real-time requirements of the applications. Therefore, how to further reduce the scale of the model parameters and improve the inference speed is one of the directions worthy of exploring. 5) Most of methods only process the geometric information (3D coordinates) of point clouds, ignoring the attribute information (color, normals, reflection intensity, etc.). Therefore, it will be of great significance to study how to simultaneously conduct quality en-hancement of geometric information and attribute information.
point cloud completion, point cloud upsampling, point cloud denoising, quality enhancement, deep learning