No-reference point cloud quality assessment based on neighbor information and attention

Chen Xiaolei; Zhang Yuru; Hu Senyong; Du Zelong

doi:10.11834/jig.230669

Mixed Reality | Views : 0 下载量: 617 CSCD: 0

PDF
Export
Share
Collection
Album

No-reference point cloud quality assessment based on neighbor information and attention
Vol. 29, Issue 10, Pages: 2979-2991(2024)
Received：28 September 2023，

Revised：2023-12-15，

Published：16 October 2024
DOI： 10.11834/jig.230669
稿件说明：

移动端阅览

陈晓雷，张育儒，胡森涌，杜泽龙. 2024. 基于邻域信息和注意力的无参考点云质量评估. 中国图象图形学报， 29(10):2979-2991 DOI： 10.11834/jig.230669.

Chen Xiaolei， Zhang Yuru， Hu Senyong， Du Zelong. 2024. No-reference point cloud quality assessment based on neighbor information and attention. Journal of Image and Graphics， 29(10):2979-2991 DOI： 10.11834/jig.230669.

摘要

目的

针对现有无参考点云质量评估方法需要将点云预处理为二维投影或其他形式导致引入额外噪声、限制空间上下文等问题，提出了一种基于邻域信息嵌入变换模块和点云级联注意力模块的无参考点云质量评估方法。

方法

将点云样本整体作为输入，减轻预处理引入的失真。使用稀疏卷积搭建U型主干网络提取多尺度特征，邻域信息嵌入变换模块逐点学习提取特征，点云级联注意力模块增强小尺度特征，提高特征信息的可辨识性，最后逐步聚合多尺度特征信息形成特征向量，经全局自适应池化和回归函数进行回归预测，得到失真点云质量分数。

结果

实验在2个数据集上与现有的12种代表性点云质量评估方法进行了比较，在SJTU-PCQA（Shanghai Jiao Tong University subjective point cloud quality assessment）数据集中，相比于性能第2的模型，PLCC（Pearson linear correlation coefficient）值提高了8.7%，SROCC（Spearman rank-order coefficient correlation）值提高了0.39%；在WPC（waterloo point cloud）数据集中，相比于性能第2的模型，PLCC值提高了4.9%，SROCC值提高了3.0%。

结论

所提出的基于邻域信息嵌入变换和级联注意力的无参考点云质量评估方法，提高了可辨识特征提取能力，使点云质量评估结果更加准确。

Abstract

Objective

This study introduces a novel method that aims to address the shortcomings of current no-reference point cloud quality assessment methods. Such methods necessitate the preprocessing of point clouds into 2D projections or other forms， which may introduce additional noise and limit the spatial contextual information of the data. The proposed approach overcomes these limitations. The approach comprises two crucial components， namely， the neighborhood information embedding transformation module and the point cloud cascading attention module. The former module is intended to capture the point cloud data’s local features and geometric structure without any extra preprocessing. This process preserves the point cloud’s original information and minimizes the potential for introducing additional noise， all while providing a more expansive spatial context. The latter module enhances the precision and flexibility of point cloud quality assessment by merging spatial and channel attention. The module dynamically learns weightings and applies them to features based on the aspects of various point cloud data， resulting in a more comprehensive understanding of multidimensional point cloud information.

Method

The proposed model employs innovative strategies to address challenges in assessing point cloud quality. In contrast to traditional approaches， it takes the original point cloud sample as input and eliminates the need for preprocessing. This process helps maintain the point cloud’s integrity and improve accuracy in assessment. Second， a U-shaped backbone network is constructed using sparse convolution to enable multiscale feature extraction， allowing the model to capture different scale features of the point cloud and understand point cloud data at local and overall levels more effectively. The module for transforming neighborhood information embedding is an essential part of the process because it extracts features through point-by-point learning. This process assists the model in thoroughly comprehending the local information present in the point cloud. Furthermore， the attention module for point cloud cascade bolsters small-scale features， elevating the recognizability of feature information. By progressively consolidating the multiscale feature information to construct a feature vector， the model can thoroughly represent the quality features of the point cloud. Ultimately， global adaptive pooling and regression functions are employed for regression prediction to finally obtain quality scores for distorted point clouds. The model’s architecture utilizes multiscale information to improve the representation and evaluation of features， resulting in increased progress and efficiency in the assessment of point cloud quality.

Result

In this study， a set of experiments was conducted to validate the efficacy of the proposed method for assessing the quality of point clouds. The results of the experiments demonstrate that the method shows substantial enhancements over 12 existing representative assessment methods for point cloud quality on two different datasets. The experiment specifically employs the SJTU-PCQA dataset， and the novel technique enhances the PLCC value by 8.7% and the SROCC value by 0.39% relative to the model with the second-highest performance. Thus， the new method more precisely evaluates the point cloud quality on the SJTU-PCQA dataset with improved correlation and performance. Similarly， the novel approach enhances the PLCC metric by 4.9% and the SROCC metric by 3.0% on the WPC dataset， surpassing the model with the second-best results. This result illustrates the efficiency and effectiveness of the new approach in point cloud quality assessment for various datasets. The results of these experiments highlight the effectiveness and superiority of the method proposed， providing substantial backing for subsequent research and applications in the arena of point cloud quality assessment. Furthermore， the method showcases its widely applicable nature.

Conclusion

The no-reference method presented in this study enhances the precision of point cloud quality assessment. The technique employs a novel structure of embedding transformation for neighborhood information and cascading attention. Emphasis is placed on recognizable feature extraction to yield more accurate results in point cloud quality assessment.

关键词

Keywords

references

Alexiou E and Ebrahimi T . 2020 . Towards a point cloud structural similarity metric // Proceedings of 2020 IEEE International Conference on Multimedia and Expo Workshops （ICMEW） . London， UK ： IEEE： 1 - 6 ［ DOI： 10.1109/ICMEW46912.2020.9106005 http://dx.doi.org/10.1109/ICMEW46912.2020.9106005 ］

Chetouani A ， Quach M ， Valenzise G and Dufaux F . 2021 . Deep learning-based quality assessment of 3D point clouds without reference // Proceedings of 2021 IEEE International Conference on Multimedia and Expo Workshops （ICMEW） . Shenzhen， China ： IEEE： 1 - 6 ［ DOI： 10.1109/ICMEW53276.2021.9455967 http://dx.doi.org/10.1109/ICMEW53276.2021.9455967 ］

Choy C ， Gwak J and Savarese S . 2019 . 4D spatio-temporal ConvNets： Minkowski convolutional neural networks // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 3070 - 3079 ［ DOI： 10.1109/CVPR.2019.00319 http://dx.doi.org/10.1109/CVPR.2019.00319 ］

Fan Y ， Zhang Z C ， Sun W ， Min X K ， Liu N ， Zhou Q ， He J ， Wang Q Y and Zhai G T . 2022 . A no-reference quality assessment metric for point cloud based on captured video sequences // Proceedings of the 24th IEEE International Workshop on Multimedia Signal Processing （MMSP） . Shanghai， China ： IEEE： 1 - 5 ［ DOI： 10.1109/MMSP55362.2022.9949359 http://dx.doi.org/10.1109/MMSP55362.2022.9949359 ］

Gong J Y ， Lou Y J ， Liu F Q ， Zhang Z W ， Chen H M ， Zhang Z Z ， Tan X ， Xie Y and Ma L Z . 2023 . Scene point cloud understanding and reconstruction technologies in 3D space . Journal of Image and Graphics ， 28 （ 6 ）： 1741 - 1766

龚靖渝，楼雨京，柳奉奇，张志伟，陈豪明，张志忠，谭鑫，谢源，马利庄 . 2023 . 三维场景点云理解与重建技术 . 中国图象图形学报， 28 （ 6 ）： 1741 - 1766 ［ DOI： 10.11834/jig.230004 http://dx.doi.org/10.11834/jig.230004 ］

Ji J Y ， Xiang K and Wang X Y . 2023 . SCVS： blind image quality assessment based on spatial correlation and visual saliency . The Visual Computer ， 39 （ 1 ）： 443 - 458 ［ DOI： 10.1007/s00371-021-02340-x http://dx.doi.org/10.1007/s00371-021-02340-x ］

Lao S S ， Gong Y ， Shi S W ， Yang S D ， Wu T H ， Wang J H ， Xia W H and Yang Y J . 2022 . Attentions help CNNs see better： attention-based hybrid image quality assessment network // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops （CVPRW） . New Orleans， USA ： IEEE： 1139 - 1148 ［ DOI： 10.1109/CVPRW56347.2022.00123 http://dx.doi.org/10.1109/CVPRW56347.2022.00123 ］

Liu Q ， Yuan H ， Su H L ， Liu H ， Wang Y ， Yang H and Hou J H . 2021 . PQA-Net： deep no reference point cloud quality assessment via multi-view projection . IEEE Transactions on Circuits and Systems for Video Technology ， 31 （ 12 ）： 4645 - 4660 ［ DOI： 10.1109/TCSVT.2021.3100282 http://dx.doi.org/10.1109/TCSVT.2021.3100282 ］

Liu Y P ， Yang Q ， Xu Y L and Yang L . 2023 . Point cloud quality assessment： dataset construction and learning-based no-reference metric . ACM Transactions on Multimedia Computing， Communications， and Applications ， 19 （ 2 S）： # 80 ［ DOI： 10.1145/3550274 http://dx.doi.org/10.1145/3550274 ］

Qi C R ， Su H ， Mo K C and Guibas L J . 2017 . PointNet： deep learning on point sets for 3D classification and segmentation // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 77 - 85 ［ DOI： 10.1109/CVPR.2017.16 http://dx.doi.org/10.1109/CVPR.2017.16 ］

Shan Z Y ， Yang Q ， Ye R ， Zhang Y J ， Xu Y L ， Xu X Z and Liu S . 2023 . GPA-Net： no-reference point cloud quality assessment with multi-task graph convolutional network . IEEE Transactions on Visualization and Computer Graphics ， 30 （ 8 ）： 4955 - 4967 ［ DOI： 10.1109/TVCG.2023.3282802 http://dx.doi.org/10.1109/TVCG.2023.3282802 ］

Sheikh H R ， Sabir M F and Bovik A C . 2006 . A statistical evaluation of recent full reference image quality assessment algorithms . IEEE Transactions on Image Processing ， 15 （ 11 ）： 3440 - 3451 ［ DOI： 10.1109/TIP.2006.881959 http://dx.doi.org/10.1109/TIP.2006.881959 ］

Simonyan K and Zisserman A . 2015 . Very deep convolutional networks for large-scale image recognition ［EB/OL］. ［ 2023-09-28 ］. https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf

Su H L ， Duanmu Z F ， Liu W T ， Liu Q and Wang Z . 2019 . Perceptual quality assessment of 3D point clouds // Proceedings of 2019 IEEE International Conference on Image Processing （ICIP） . Taipei， China ： IEEE： 3182 - 3186 ［ DOI： 10.1109/ICIP.2019.8803298 http://dx.doi.org/10.1109/ICIP.2019.8803298 ］

Tian D ， Ochimizu H ， Feng C ， et al . 2017a . Evaluation metrics for point cloud compression. ISO/IEC JTC m74008， Geneva ， Switzerland ， 1 （ 3 ）

Tian D ， Ochimizu H ， Feng C ， et al . 2017b . Updates and integration of evaluation metric software for PCC. ISO/IEC JTC1/SC29/WG11 input document MPEG 2017 M， 40522 ： 26

Tliba M ， Chetouani A ， Valenzise G and Dufaux F . 2022 . Representation learning optimization for 3D point cloud quality assessment without reference // Proceedings of 2022 IEEE International Conference on Image Processing （ICIP） . Bordeaux， France ： IEEE： 3702 - 3706 ［ DOI： 10.1109/ICIP46576.2022.9897689 http://dx.doi.org/10.1109/ICIP46576.2022.9897689 ］

Tliba M ， Chetouani A ， Valenzise G and Dufaux F . 2023 . Quality evaluation of point clouds： a novel no-reference approach using Transformer-based architecture ［EB/OL］. ［ 2023-03-15 ］. https://arxiv.org/pdf/2303.08634.pdf https://arxiv.org/pdf/2303.08634.pdf

Viola I and Cesar P . 2020 . A reduced reference metric for visual quality evaluation of point cloud contents . IEEE Signal Processing Letters ， 27 ： 1660 - 1664 ［ DOI： 10.1109/LSP.2020.3024065 http://dx.doi.org/10.1109/LSP.2020.3024065 ］

Yang Q ， Chen H ， Ma Z ， Xu Y L ， Tang R J and Sun J . 2021 . Predicting the perceptual quality of point cloud： a 3D-to-2D projection-based exploration . IEEE Transactions on Multimedia ， 23 ： 3877 - 3891 ［ DOI： 10.1109/TMM.2020.3033117 http://dx.doi.org/10.1109/TMM.2020.3033117 ］

Yang Q ， Ma Z ， Xu Y L ， Li Z and Sun J . 2022a . Inferring point cloud quality via graph similarity . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 44 （ 6 ）： 3015 - 3029 ［ DOI： 10.1109/TPAMI.2020.3047083 http://dx.doi.org/10.1109/TPAMI.2020.3047083 ］

Yang Q ， Liu Y P ， Chen S H ， Xu Y L and Sun J . 2022b . No-reference point cloud quality assessment via domain adaptation // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 21147 - 21156 ［ DOI： 10.1109/CVPR52688.2022.02050 http://dx.doi.org/10.1109/CVPR52688.2022.02050 ］

You J Y and Korhonen J . 2021 . Transformer for image quality assessment // Proceedings of 2021 IEEE International Conference on Image Processing （ICIP） . Anchorage， USA ： IEEE： 1389 - 1393 ［ DOI： 10.1109/ICIP42928.2021.9506075 http://dx.doi.org/10.1109/ICIP42928.2021.9506075 ］

Zhang Z C ， Sun W ， Min X K ， Wang T ， Lu W and Zhai G T . 2022 . No-reference quality assessment for 3D colored point cloud and mesh models . IEEE Transactions on Circuits and Systems for Video Technology ， 32 （ 11 ）： 7618 - 7631 ［ DOI： 10.1109/TCSVT.2022.3186894 http://dx.doi.org/10.1109/TCSVT.2022.3186894 ］

Zhang Z C ， Sun W ， Zhu Y C ， Min X K ， Wu W ， Chen Y and Zhai G T . 2023 . Evaluating point cloud from moving camera videos： a no-reference metric . IEEE Transactions on Multimedia ， 1 - 13 ［ DOI： 10.1109/TMM.2023.3340894 http://dx.doi.org/10.1109/TMM.2023.3340894 ］

Zhu W ， Zhang Y H ， Ying Y ， Zheng Y Y and He D F . 2023 . A dense residual structure and multi-scale pruning-relevant point cloud compression network . Journal of Image and Graphics ， 28 （ 7 ）： 2105 - 2119

朱威，张雨航，应悦，郑雅羽，何德峰 . 2023 . 结合密集残差结构和多尺度剪枝的点云压缩网络 . 中国图象图形学报， 28 （ 7 ）： 2105 - 2119 ［ DOI： 10.11834/jig.220047 http://dx.doi.org/10.11834/jig.220047 ］