虚拟现实图像客观质量评价研究进展

周玉; 汪一; 李雷达; 高陈强; 卢兆林

doi:10.11834/jig.210949

综述 | 浏览量 : 0 下载量: 410 CSCD: 1

PDF
导出
分享
收藏
专辑

虚拟现实图像客观质量评价研究进展
Research progress in objective quality evaluation of virtual reality images
2022年27卷第8期页码：2313-2328
收稿：2021-10-13，

修回：2021-12-15，

录用：2021-12-22，

纸质出版：2022-08-16
DOI： 10.11834/jig.210949
稿件说明：

移动端阅览

周玉, 汪一, 李雷达, 高陈强, 卢兆林. 虚拟现实图像客观质量评价研究进展[J]. 中国图象图形学报, 2022,27(8):2313-2328. DOI： 10.11834/jig.210949.

Yu Zhou, Yi Wang, Leida Li, Chenqiang Gao, Zhaolin Lu. Research progress in objective quality evaluation of virtual reality images[J]. Journal of Image and Graphics, 2022, 27(8): 2313-2328. DOI： 10.11834/jig.210949.

摘要

随着科学技术的发展，虚拟现实(virtual reality

VR)技术逐渐渗透到医疗、教育、军事和娱乐等众多领域，并凭借在各个领域广阔的应用前景而受到广泛关注。鉴于视觉质量是决定VR技术能否成功应用的关键，且图像是VR应用最基础和最重要的视觉信息载体，VR图像质量评价已经成为质量评价领域的重要前沿性研究方向。与传统图像质量评价类似，VR图像质量评价可以分为主观质量评价和客观质量评价。由于客观质量评价相比主观质量评价具有成本低、稳定性高和应用范围广等优点，对VR图像客观质量评价的研究受到了国内外学者的高度重视。目前，关于VR图像客观质量评价的研究已经取得了一定进展，但是文献中缺少对该方向相关研究方法的综述。基于此，本文针对VR图像客观质量评价的研究进行概述。首先，对VR图像质量评价的研究现状进行分析。然后，重点对现有的VR图像客观质量评价模型进行综述。具体地，根据模型是否需要使用原始无失真图像信息作为参考，将现有的VR图像客观质量评价模型划分为全参考型和无参考型两大类。其中，全参考型方法进一步划分为基于峰值信噪比/结构相似度的方法和基于传统机器学习的方法。根据特征表达空间的不同，无参考型VR图像质量评价模型划分为3类：基于等矩形投影表达空间的方法、基于其他投影表达空间的方法和基于实际观看空间的方法。介绍完各类模型后，分别对其相应的优缺点进行分析。此外，本文对VR图像客观质量评价模型的性能评价指标和现有VR图像质量评价数据库进行了归纳。最后对VR图像客观质量评价模型的应用进行了介绍，并指出了未来的研究可能的发展方向。

Abstract

Virtual reality (VR) technology has gradually penetrated into many fields such as medical education

military and entertainment. Given visual quality is the key to the successful application of VR technology

and the image is visual information benched carrier of VR applications

VR image quality evaluation has become an important frontier research direction for quality evaluation. Just like traditional image quality evaluation

VR image quality evaluation can be divided into subjective quality evaluation and objective quality evaluation. Among them

the subjective quality evaluation methods refer to images scoring through human eyes followed by some data processing steps to obtain the subjective scores

while the objective quality evaluation methods focus on the methods of images scoring based on designing the mathematical model using computers to simulate the subjective scoring results as closely as possible. Compared to the subjective quality evaluation

the objective quality evaluation has its priorities s of lower cost

stronger stability and wider application scope. Many researchers in the scientific research institutions and colleges have dedicated to studying the objective quality evaluation of VR images. Our executive summary is focused on the research of objective quality evaluation of VR images. First

the current situation of VR image quality evaluation is summarized. Then

the existing objective quality evaluation models of VR images are mainly introduced and analyzed. According to whether the models need to use the original undistorted image information as modeling reference

current objective quality evaluation models for VR images are divided into two types

including the full-reference (FR) and no-reference (NR) types. The ground truth based FR models need the completed original image information as reference

while the NR models can achieve quality evaluation of a distorted VR image without any reference information. Specifically

the FR models can be divided into two categories for VR images

including the peak-signal-to-noise ratio/structural similarity (PSNR/SSIM) based methods and the machine learning based methods. The latter first extract features from VR images and then train the quality evaluation model via the support vector regression method or the random forest method. The NR models are further divided into three categories: the equirectangular projection (ERP) expression space based methods

the other projection expression spaces based methods and the actual viewing space based methods. These models are classified according to the space in which features are extracted. For the first kind of models

the raw spherical VR image is first transferred to the ERP space for feature expression

while it is converted to other projection spaces or the actual viewing space for the other two kinds of methods. The reason why the ERP expression space based methods are listed as a separate category is that the ERP space is the default and the mostly used projection space. Followed the space transformation

there are also two options for the sequential quality evaluation consisting of the traditional quality evaluation methods and the deep-learning based methods. For the traditional ones

the manual features are first extracted and fused to generate the final quality score or fed into the model trainer to obtain the quality assessment model. For the deep-learning based methods

both the feature extraction and quality prediction steps are conducted based on deep neural networks. The pros and cons of them are further analyzed. In addition

the performance evaluation indexes of the objective quality evaluation models of VR images are introduced

which are consistent to other image quality evaluations. The detailed existing VR image databases are summarized subsequently. Finally

our review focuses on the applications of VR image objective quality evaluation models and predicts the future potentials further.

关键词

Keywords

references

Athar S and Wang Z. 2019. A comprehensive performance evaluation of image quality assessment algorithms. IEEE Access, 7: 140030-140070 [DOI: 10.1109/ACCESS.2019.2943319]

Cao F. 2019. Overview of VR technology research and development at home and abroad. China Science and Technology Information, (5): 36-37

曹凡. 2019. 国内外VR技术研发现状综述. 中国科技信息, (5): 36-37 [DOI: 10.3969/j.issn.1001-8972.2019.05.010]

Cao L Y, Jiang G Y, Jiang Z D, Yu M, Qi Y B and Ho Y S. 2021. Quality measurement for high dynamic range omnidirectional image systems. IEEE Transactions on Instrumentation and Measurement, 70: #5012415 [DOI: 10.1109/TIM.2021.3093940]

Chang C H, Sato Y and Chuang Y Y. 2014. Shape-preserving half-projective warps for image stitching//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Columbus, USA: IEEE: 3254-3261 [ DOI: 10.1109/CVPR.2014.422 http://dx.doi.org/10.1109/CVPR.2014.422 ]

Chen M X, Jin Y Z, Goodall T, Yu X X and Bovik A C. 2020. Study of 3D virtual reality picture quality. IEEE Journal of Selected Topics in Signal Processing, 14(1): 89-102 [DOI: 10.1109/JSTSP.2019.2956408]

Chen S J, Zhang Y X, Li Y M, Chen Z Z and Wang Z. 2018. Spherical structural similarity index for objective omnidirectional video quality assessment//Proceedings of 2018 IEEE International Conference on Multimedia and Expo (ICME). San Diego, USA: IEEE: 1-6 [ DOI: 10.1109/ICME.2018.8486584 http://dx.doi.org/10.1109/ICME.2018.8486584 ]

Duan H Y, Zhai G T, Min X K, Zhu Y C, Fang Y and Yang X K. 2018. Perceptual quality assessment of omnidirectional images//2018 IEEE International Symposium on Circuits and Systems (ISCAS). Florence, Italy: IEEE: 1-5 [ DOI: 10.1109/ISCAS.2018.8351786 http://dx.doi.org/10.1109/ISCAS.2018.8351786 ]

Duan H Y, Zhai G T, Yang X K, Li D and Zhu W H. 2017. IVQAD 2017: an immersive video quality assessment database//Proceedings of 2017 International Conference on Systems, Signals and Image Processing (IWSSIP). Poznan, Poland: IEEE: 1-5 [ DOI: 10.1109/IWSSIP.2017.7965610 http://dx.doi.org/10.1109/IWSSIP.2017.7965610 ]

Fang Y M, Sui X J, Yan J B, Liu X L and Huang L P. 2021. Progress in no-reference image quality assessment. Journal of Image and Graphics, 26(2): 265-286

方玉明, 眭相杰, 鄢杰斌, 刘学林, 黄丽萍. 2021. 无参考图像质量评价研究进展. 中国图象图形学报, 26(2): 265-286 [DOI: 10.11834/jig.200274]

Fang Y M, Yan J B, Du R G, Zuo Y F, Wen W Y, Zeng Y and Li L D. 2020. Blind quality assessment for tone-mapped images by analysis of gradient and chromatic statistics. IEEE Transactions on Multimedia, 23: 955-966 [DOI: 10.1109/TMM.2020.2991528]

Fang Y M, Yan J B, Li L D, Wu J J and Lin W S. 2018. No reference quality assessment for screen content images with both local and global feature representation. IEEE Transactions on Image Processing, 27(4): 1600-1610 [DOI: 10.1109/TIP.2017.2781307]

Fang Y M, Yan J B, Liu J Y, Wang S Q, Li Q H and Guo Z M. 2017. Objective quality assessment of screen content images by uncertainty weighting. IEEE Transactions on Image Processing, 26(4): 2016-2027 [DOI: 10.1109/TIP.2017.2669840]

Gu K. 2015. Research on Perceptual and Statistical Image Quality Assessment and Its Application. Shanghai: Shanghai Jiao Tong University

顾锞. 2015. 基于感知和统计模型的图像质量评价技术及应用研究. 上海: 上海交通大学

Jiang G Y, Song H, Yu M, Song Y and Peng Z J. 2018. Blind tone-mapped image quality assessment based on brightest/darkest regions, naturalness and aesthetics. IEEE Access, 6: 2231-2240 [DOI: 10.1109/ACCESS.2017.2782320]

Jiang H, Jiang G Y, Yu M, Zhang Y, Yang Y, Peng Z J, Chen F and Zhang Q B. 2021a. Cubemap-based perception-driven blind quality assessment for 360-degree images. IEEE Transactions on Image Processing, 30: 2364-2377 [DOI: 10.1109/TIP.2021.3052073]

Jiang Q P, Peng Z Y, Yue G H, Li H and Shao F. 2021b. No-reference image contrast evaluation by generating bidirectional pseudo references. IEEE Transactions on Industrial Informatics, 17(9): 6062-6072 [DOI: 10.1109/TII.2020.3035448]

Kim H G, Lim H T and Ro Y M. 2020. Deep virtual reality image quality assessment with human perception guider for omnidirectional image. IEEE Transactions on Circuits and Systems for Video Technology, 30(4): 917-928 [DOI: 10.1109/TCSVT.2019.2898732]

Li C, Xu M, Du X Z and Wang Z L. 2018. Bridge the gap between VQA and human behavior on omnidirectional video: a large-scale dataset and a deep learning model//Proceedings of the 26th ACM Multimedia Conference. Seoul, Korea(South): ACM: 932-940 [ DOI: 10.1145/3240508.3240581 http://dx.doi.org/10.1145/3240508.3240581 ]

Li J, Zhao Y F, Ye W H, Yu K W and Ge S M. 2020a. Attentive deep stitching and quality assessment for 360° omnidirectional images. IEEE Journal of Selected Topics in Signal Processing, 14(1): 209-221 [DOI: 10.1109/JSTSP.2019.2953950]

Li L D, Zhou Y, Gu K, Yang Y Z and Fang Y M. 2020b. Blind realistic blur assessment based on discrepancy learning. IEEE Transactions on Circuits and Systems for Video Technology, 30(11): 3859-3869 [DOI: 10.1109/TCSVT.2019.2947450]

Li L D, Zhou Y, Wu J J, Li F and Shi G M. 2021. Quality index for view synthesis by measuring instance degradation and global appearance. IEEE Transactions on Multimedia, 23: 320-332 [DOI: 10.1109/TMM.2020.2980185]

Lim H T, Kim H G and Ra Y M. 2018. VR IQA NET: deep virtual reality image quality assessment using adversarial learning//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary, Canada: IEEE: 6737-6741 [ DOI: 10.1109/ICASSP.2018.8461317 http://dx.doi.org/10.1109/ICASSP.2018.8461317 ]

Liu Y, Yu H W, Huang B Q, Yue G H and Song B Y. 2021. Blind omnidirectional image quality assessment based on structure and natural features. IEEE Transactions on Instrumentation and Measurement, 70: #5014011 [DOI: 10.1109/TIM.2021.3102691]

Ma K D, Liu W T, Liu T L, Wang Z and Tao D C. 2017. DipIQ: blind image quality assessment by learning-to-rank discriminable image pairs. IEEE Transactions on Image Processing, 26(8): 3951-3964 [DOI: 10.1109/TIP.2017.2708503]

Ma K D, Liu W T, Zhang K, Duanmu Z F, Wang Z and Zuo W M. 2018. End-to-end blind image quality assessment using deep neural networks. IEEE Transactions on Image Processing, 27(3): 1202-1213 [DOI: 10.1109/TIP.2017.2774045]

Madhusudana P C and Soundararajan R. 2019. Subjective and objective quality assessment of stitched images for virtual reality. IEEE Transactions on Image Processing, 28(11): 5620-5635 [DOI: 10.1109/TIP.2019.2921858]

Min X K, Zhai G T, Gu K, Liu Y T and Yang X K. 2018. Blind image quality estimation via distortion aggravation. IEEE Transactions on Broadcasting, 64(2): 508-517 [DOI: 10.1109/TBC.2018.2816783]

Min X K, Zhai G T, Zhou J T, Farias M C Q and Bovik A C. 2020. Study of subjective and objective quality assessment of audio-visual signals. IEEE Transactions on Image Processing, 29: 6054-6068 [DOI: 10.1109/TIP.2020.2988148]

Moorthy A K and Bovik A C. 2010. A two-step framework for constructing blind image quality indices. IEEE Signal Processing Letters, 17(5): 513-516 [DOI: 10.1109/LSP.2010.2043888]

Moorthy A K and Bovik A C. 2011. Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Transactions on Image Processing, 20(12): 3350-3364 [DOI: 10.1109/TIP.2011.2147325]

Nie L, Lin C Y, Liao K, Liu S C and Zhao Y. 2021. Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Transactions on Image Processing, 30: 6184-6197 [DOI: 10.1109/TIP.2021.3092828]

Niu Y Z, Zhong Y N, Guo W Z, Shi Y Q and Chen P K. 2019. 2D and 3D image quality assessment: a survey of metrics and challenges. IEEE Access, 7: 782-801 [DOI: 10.1109/ACCESS.2018.2885818]

Shao F, Fu Z Q, Jiang Q P, Jiang G Y and Ho Y S. 2021. Transformation-aware similarity measurement for image retargeting quality assessment via bidirectional rewarping. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(5): 3053-3067 [DOI: 10.1109/TSMC.2019.2917496]

Sui X J, Ma K D, Yao Y R and Fang Y M. 2021. Perceptual quality assessment of omnidirectional images as moving cam era videos. IEEE Transactions on Visualization and Computer Graphics: #3050888 [ DOI: 10.1109/TVCG.2021.3050888 http://dx.doi.org/10.1109/TVCG.2021.3050888 ]

Sun W, Gu K, Zhai G T, Ma S W, Lin W S and Calle P L. 2017a. CVIQD: subjective quality evaluation of compressed virtual reality images//Proceedings of 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China: IEEE: 3450-3454 [ DOI: 10.1109/ICIP.2017.8296923 http://dx.doi.org/10.1109/ICIP.2017.8296923 ]

Sun W, Min X K, Zhai G T, Gu K, Duan H Y and Ma S W. 2020. MC360IQA: a multi-channel CNN for blind 360-degree image quality assessment. IEEE Journal of Selected Topics in Signal Processing, 14(1): 64-77 [DOI: 10.1109/JSTSP.2019.2955024]

Sun Y L, Lu A and Yu L. 2017b. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE Signal Processing Letters, 24(9): 1408-1412 [DOI: 10.1109/LSP.2017.2720693]

Video Quality Experts Group (VQEG). 2000. Final report from the video quality experts group on the validation of objective models of video quality assessment [EB/OL ] . [2020-07-08 ] . http://videoclarity.com/PDF/COM-80E_final_report.pdf http://videoclarity.com/PDF/COM-80E_final_report.pdf

Wang S Q, Gu K, Ma S W and Gao W. 2016. Joint chroma downsampling and upsampling for screen content image. IEEE Transactions on Circuits and Systems for Video Technology, 26(9): 1595-1609 [DOI: 10.1109/TCSVT.2015.2461891]

Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612 [DOI: 10.1109/TIP.2003.819861]

Wang Z M. 2015. Review of no-reference image quality assessment. Acta Automatica Sinica, 41(6): 1062-1079

王志明. 2015. 无参考图像质量评价综述. 自动化学报, 41(6): 1062-1079 [DOI: 10.16383/j.aas.2015.c140404]

Wu J J. 2014. Image Information Perception and Quality Assessment based on the Human Visual System. Xi′an: Xidian University

吴金建. 2014. 基于人类视觉系统的图像信息感知和图像质量评价. 西安: 西安电子科技大学

Wu J J, Ma J P, Liang F H, Dong W S, Shi G M and Lin W S. 2020. End-to-end blind image quality prediction with cascaded deep neural network. IEEE Transactions on Image Processing, 29: 7414-7426 [DOI: 10.1109/TIP.2020.3002478]

Wu Q B, Li H L, Meng F M and Ngan K N. 2018. Toward a blind quality metric for temporally distorted streaming video. IEEE Transactions on Broadcasting, 64(2): 367-378 [DOI: 10.1109/TBC.2017.2786023]

Xu W and Mulligan J. 2010. Performance evaluation of color correction approaches for automatic multi-view image and video stitching//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, USA: IEEE: 263-270 [ DOI: 10.1109/CVPR.2010.5540202 http://dx.doi.org/10.1109/CVPR.2010.5540202 ]

Xu J H, Zhou W and Chen Z B. 2021. Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks. IEEE Transactions on Circuits and Systems for Video Technology, 31(5): 1724-1737 [DOI: 10.1109/TCSVT.2020.3015186]

Xu M, Li C, Liu Y F, Deng X and Lu J X. 2017. A subjective visual quality assessment method of panoramic videos//Proceedings of 2017 IEEE International Conference on Multimedia and Expo (ICME). Hong Kong, China: IEEE: 517-522 [ DOI: 10.1109/ICME.2017.8019351 http://dx.doi.org/10.1109/ICME.2017.8019351 ]

Yang J C, Liu T L, Jiang B, Song H B and Lu W. 2018. 3D panoramic virtual reality video quality assessment based on 3D convolutional neural networks. IEEE Access, 6: 38669-38682 [DOI: 10.1109/ACCESS.2018.2854922]

Yang X H, Li F and Liu H T. 2019. Asurvey of DNN methods for blind image quality assessment. IEEE Access, 7: 123788-123806 [DOI: 10.1109/ACCESS.2019.2938900]

Yuan Y T, Fang F M and Zhang G X. 2021. Superpixel-based seamless image stitching for UAV images. IEEE Transactions on Geoscience and Remote Sensing, 59(2): 1565-1576 [DOI: 10.1109/TGRS.2020.2999404]

Yu M, Lakshman H and Girod B. 2015. A framework to evaluate omnidirectional video coding schemes//Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality. Fukuoka, Japan: IEEE: 31-36 [ DOI: 10.1109/ISMAR.2015.12 http://dx.doi.org/10.1109/ISMAR.2015.12 ]

Yue G H, Hou C P and Zhou T W. 2019. Blind quality assessment of tone-mapped images considering colorfulness, naturalness, and structure. IEEE Transactions on Industrial Electronics, 66(5): 3784-3793 [DOI: 10.1109/TIE.2018.2851984]

Zakharchenko V, Choi K P and Park J H. 2016. Quality metric for spherical panoramic video//Proceedings Volume 9970, Optics and Photonics for Information Processing X. San Diego, USA: SPIE: 1-9 [ DOI: 10.1117/12.2235885 http://dx.doi.org/10.1117/12.2235885 ]

Zhai G T and Min X K. 2020. Perceptual image quality assessment: a survey. Science China Information Sciences, 63(11): #211301 [DOI: 10.1007/s11432-019-2757-1]

Zhan Y B, Zhang R and Wu Q. 2017. A structural variation classification model for image quality assessment. IEEE Transactions on Multimedia, 19(8): 1837-1847 [DOI: 10.1109/TMM.2017.2689923]

Zhang L, Zhang L and Bovik A C. 2015. A feature-enriched completely blind image quality evaluator. IEEE Transactions onImage Processing, 24(8): 2579-2591 [DOI: 10.1109/TIP.2015.2426416]

Zhang W X, Ma K D, Yan J, Deng D X and Wang Z. 2020. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 30(1): 36-47 [DOI: 10.1109/TCSVT.2018.2886771]

Zheng F. 2020. Design and Research on Real-Time AI Reporting of VR E-Sports News. Harbin: Harbin Institute of Technology

郑菲. 2020. 虚拟现实电竞新闻实时智能报道的设计研究. 哈尔滨: 哈尔滨工业大学

Zheng K K. 2017. Research on Stitching Quality Assessment Method of Virtual Reality Panoramic Video. Xiamen: Xiamen University

郑珂珂. 2017. VR全景视频的拼接质量评价方法研究. 厦门: 厦门大学

Zheng X L, Jiang G Y, Yu M and Jiang H. 2020. Segmented spherical projection-based blind omnidirectional image quality assessment. IEEE Access, 8: 31647-31659 [DOI: 10.1109/ACCESS.2020.2972158]

Zhou Y. 2019. Study on the Objective Quality Assessment Metrics of Virtual View Synthesis. Xuzhou: China University of Mining and Technology

周玉. 2019. 面向虚拟视角合成的客观质量评价方法研究. 徐州: 中国矿业大学

Zhou Y, Li L D, Wang S Q, Wu J J, Fang Y M and Gao X B. 2019. No-reference quality assessment for view synthesis using DoG-based edge statistics and texture naturalness. IEEE Transactions on Image Processing, 28(9): 4566-4579 [DOI: 10.1109/TIP.2019.2912463]

Zhou Y, Li L D, Wu J J, Gu K, Dong W S and Shi G M. 2018. Blind quality index for multiply distorted images using biorder structure degradation and nonlocal statistics. IEEE Transactions on Multimedia, 20(11): 3019-3032 [DOI: 10.1109/TMM.2018.2829607]

Zhou Y, Sun Y J, Li L D, Gu K and Fang Y M. 2021. Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network. IEEE Transactions on Circuits and Systems for Video Technology, 32: 1767-1777[DOI: 10.1109/TCSVT.2021.3081162]

Zhou Z, Zhou Y and Xiao J J. 2015. Survey on augmented virtual environment and augmented reality. Scientia Sinica Informationis, 45(2): 157-180

周忠, 周颐, 肖江剑. 2015. 虚拟现实增强技术综述. 中国科学: 信息科学, 45(2): 157-180 [DOI: 10.1360/N112014-00076]