浏览全部资源
扫码关注微信
Received：13 October 2021，
Revised：2021-12-15，
Accepted：22 December 2021，
Published：16 August 2022
移动端阅览
随着科学技术的发展，虚拟现实(virtual reality
VR)技术逐渐渗透到医疗、教育、军事和娱乐等众多领域，并凭借在各个领域广阔的应用前景而受到广泛关注。鉴于视觉质量是决定VR技术能否成功应用的关键，且图像是VR应用最基础和最重要的视觉信息载体，VR图像质量评价已经成为质量评价领域的重要前沿性研究方向。与传统图像质量评价类似，VR图像质量评价可以分为主观质量评价和客观质量评价。由于客观质量评价相比主观质量评价具有成本低、稳定性高和应用范围广等优点，对VR图像客观质量评价的研究受到了国内外学者的高度重视。目前，关于VR图像客观质量评价的研究已经取得了一定进展，但是文献中缺少对该方向相关研究方法的综述。基于此，本文针对VR图像客观质量评价的研究进行概述。首先，对VR图像质量评价的研究现状进行分析。然后，重点对现有的VR图像客观质量评价模型进行综述。具体地，根据模型是否需要使用原始无失真图像信息作为参考，将现有的VR图像客观质量评价模型划分为全参考型和无参考型两大类。其中，全参考型方法进一步划分为基于峰值信噪比/结构相似度的方法和基于传统机器学习的方法。根据特征表达空间的不同，无参考型VR图像质量评价模型划分为3类：基于等矩形投影表达空间的方法、基于其他投影表达空间的方法和基于实际观看空间的方法。介绍完各类模型后，分别对其相应的优缺点进行分析。此外，本文对VR图像客观质量评价模型的性能评价指标和现有VR图像质量评价数据库进行了归纳。最后对VR图像客观质量评价模型的应用进行了介绍，并指出了未来的研究可能的发展方向。
Virtual reality (VR) technology has gradually penetrated into many fields such as medical education
military and entertainment. Given visual quality is the key to the successful application of VR technology
and the image is visual information benched carrier of VR applications
VR image quality evaluation has become an important frontier research direction for quality evaluation. Just like traditional image quality evaluation
VR image quality evaluation can be divided into subjective quality evaluation and objective quality evaluation. Among them
the subjective quality evaluation methods refer to images scoring through human eyes followed by some data processing steps to obtain the subjective scores
while the objective quality evaluation methods focus on the methods of images scoring based on designing the mathematical model using computers to simulate the subjective scoring results as closely as possible. Compared to the subjective quality evaluation
the objective quality evaluation has its priorities s of lower cost
stronger stability and wider application scope. Many researchers in the scientific research institutions and colleges have dedicated to studying the objective quality evaluation of VR images. Our executive summary is focused on the research of objective quality evaluation of VR images. First
the current situation of VR image quality evaluation is summarized. Then
the existing objective quality evaluation models of VR images are mainly introduced and analyzed. According to whether the models need to use the original undistorted image information as modeling reference
current objective quality evaluation models for VR images are divided into two types
including the full-reference (FR) and no-reference (NR) types. The ground truth based FR models need the completed original image information as reference
while the NR models can achieve quality evaluation of a distorted VR image without any reference information. Specifically
the FR models can be divided into two categories for VR images
including the peak-signal-to-noise ratio/structural similarity (PSNR/SSIM) based methods and the machine learning based methods. The latter first extract features from VR images and then train the quality evaluation model via the support vector regression method or the random forest method. The NR models are further divided into three categories: the equirectangular projection (ERP) expression space based methods
the other projection expression spaces based methods and the actual viewing space based methods. These models are classified according to the space in which features are extracted. For the first kind of models
the raw spherical VR image is first transferred to the ERP space for feature expression
while it is converted to other projection spaces or the actual viewing space for the other two kinds of methods. The reason why the ERP expression space based methods are listed as a separate category is that the ERP space is the default and the mostly used projection space. Followed the space transformation
there are also two options for the sequential quality evaluation consisting of the traditional quality evaluation methods and the deep-learning based methods. For the traditional ones
the manual features are first extracted and fused to generate the final quality score or fed into the model trainer to obtain the quality assessment model. For the deep-learning based methods
both the feature extraction and quality prediction steps are conducted based on deep neural networks. The pros and cons of them are further analyzed. In addition
the performance evaluation indexes of the objective quality evaluation models of VR images are introduced
which are consistent to other image quality evaluations. The detailed existing VR image databases are summarized subsequently. Finally
our review focuses on the applications of VR image objective quality evaluation models and predicts the future potentials further.
Athar S and Wang Z. 2019. A comprehensive performance evaluation of image quality assessment algorithms. IEEE Access, 7: 140030-140070 [DOI: 10.1109/ACCESS.2019.2943319]
Cao F. 2019. Overview of VR technology research and development at home and abroad. China Science and Technology Information, (5): 36-37
曹凡. 2019. 国内外VR技术研发现状综述. 中国科技信息, (5): 36-37 [DOI: 10.3969/j.issn.1001-8972.2019.05.010]
Cao L Y, Jiang G Y, Jiang Z D, Yu M, Qi Y B and Ho Y S. 2021. Quality measurement for high dynamic range omnidirectional image systems. IEEE Transactions on Instrumentation and Measurement, 70: #5012415 [DOI: 10.1109/TIM.2021.3093940]
Chang C H, Sato Y and Chuang Y Y. 2014. Shape-preserving half-projective warps for image stitching//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Columbus, USA: IEEE: 3254-3261 [ DOI: 10.1109/CVPR.2014.422 http://dx.doi.org/10.1109/CVPR.2014.422 ]
Chen M X, Jin Y Z, Goodall T, Yu X X and Bovik A C. 2020. Study of 3D virtual reality picture quality. IEEE Journal of Selected Topics in Signal Processing, 14(1): 89-102 [DOI: 10.1109/JSTSP.2019.2956408]
Chen S J, Zhang Y X, Li Y M, Chen Z Z and Wang Z. 2018. Spherical structural similarity index for objective omnidirectional video quality assessment//Proceedings of 2018 IEEE International Conference on Multimedia and Expo (ICME). San Diego, USA: IEEE: 1-6 [ DOI: 10.1109/ICME.2018.8486584 http://dx.doi.org/10.1109/ICME.2018.8486584 ]
Duan H Y, Zhai G T, Min X K, Zhu Y C, Fang Y and Yang X K. 2018. Perceptual quality assessment of omnidirectional images//2018 IEEE International Symposium on Circuits and Systems (ISCAS). Florence, Italy: IEEE: 1-5 [ DOI: 10.1109/ISCAS.2018.8351786 http://dx.doi.org/10.1109/ISCAS.2018.8351786 ]
Duan H Y, Zhai G T, Yang X K, Li D and Zhu W H. 2017. IVQAD 2017: an immersive video quality assessment database//Proceedings of 2017 International Conference on Systems, Signals and Image Processing (IWSSIP). Poznan, Poland: IEEE: 1-5 [ DOI: 10.1109/IWSSIP.2017.7965610 http://dx.doi.org/10.1109/IWSSIP.2017.7965610 ]
Fang Y M, Sui X J, Yan J B, Liu X L and Huang L P. 2021. Progress in no-reference image quality assessment. Journal of Image and Graphics, 26(2): 265-286
方玉明, 眭相杰, 鄢杰斌, 刘学林, 黄丽萍. 2021. 无参考图像质量评价研究进展. 中国图象图形学报, 26(2): 265-286 [DOI: 10.11834/jig.200274]
Fang Y M, Yan J B, Du R G, Zuo Y F, Wen W Y, Zeng Y and Li L D. 2020. Blind quality assessment for tone-mapped images by analysis of gradient and chromatic statistics. IEEE Transactions on Multimedia, 23: 955-966 [DOI: 10.1109/TMM.2020.2991528]
Fang Y M, Yan J B, Li L D, Wu J J and Lin W S. 2018. No reference quality assessment for screen content images with both local and global feature representation. IEEE Transactions on Image Processing, 27(4): 1600-1610 [DOI: 10.1109/TIP.2017.2781307]
Fang Y M, Yan J B, Liu J Y, Wang S Q, Li Q H and Guo Z M. 2017. Objective quality assessment of screen content images by uncertainty weighting. IEEE Transactions on Image Processing, 26(4): 2016-2027 [DOI: 10.1109/TIP.2017.2669840]
Gu K. 2015. Research on Perceptual and Statistical Image Quality Assessment and Its Application. Shanghai: Shanghai Jiao Tong University
顾锞. 2015. 基于感知和统计模型的图像质量评价技术及应用研究. 上海: 上海交通大学
Jiang G Y, Song H, Yu M, Song Y and Peng Z J. 2018. Blind tone-mapped image quality assessment based on brightest/darkest regions, naturalness and aesthetics. IEEE Access, 6: 2231-2240 [DOI: 10.1109/ACCESS.2017.2782320]
Jiang H, Jiang G Y, Yu M, Zhang Y, Yang Y, Peng Z J, Chen F and Zhang Q B. 2021a. Cubemap-based perception-driven blind quality assessment for 360-degree images. IEEE Transactions on Image Processing, 30: 2364-2377 [DOI: 10.1109/TIP.2021.3052073]
Jiang Q P, Peng Z Y, Yue G H, Li H and Shao F. 2021b. No-reference image contrast evaluation by generating bidirectional pseudo references. IEEE Transactions on Industrial Informatics, 17(9): 6062-6072 [DOI: 10.1109/TII.2020.3035448]
Kim H G, Lim H T and Ro Y M. 2020. Deep virtual reality image quality assessment with human perception guider for omnidirectional image. IEEE Transactions on Circuits and Systems for Video Technology, 30(4): 917-928 [DOI: 10.1109/TCSVT.2019.2898732]
Li C, Xu M, Du X Z and Wang Z L. 2018. Bridge the gap between VQA and human behavior on omnidirectional video: a large-scale dataset and a deep learning model//Proceedings of the 26th ACM Multimedia Conference. Seoul, Korea(South): ACM: 932-940 [ DOI: 10.1145/3240508.3240581 http://dx.doi.org/10.1145/3240508.3240581 ]
Li J, Zhao Y F, Ye W H, Yu K W and Ge S M. 2020a. Attentive deep stitching and quality assessment for 360° omnidirectional images. IEEE Journal of Selected Topics in Signal Processing, 14(1): 209-221 [DOI: 10.1109/JSTSP.2019.2953950]
Li L D, Zhou Y, Gu K, Yang Y Z and Fang Y M. 2020b. Blind realistic blur assessment based on discrepancy learning. IEEE Transactions on Circuits and Systems for Video Technology, 30(11): 3859-3869 [DOI: 10.1109/TCSVT.2019.2947450]
Li L D, Zhou Y, Wu J J, Li F and Shi G M. 2021. Quality index for view synthesis by measuring instance degradation and global appearance. IEEE Transactions on Multimedia, 23: 320-332 [DOI: 10.1109/TMM.2020.2980185]
Lim H T, Kim H G and Ra Y M. 2018. VR IQA NET: deep virtual reality image quality assessment using adversarial learning//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary, Canada: IEEE: 6737-6741 [ DOI: 10.1109/ICASSP.2018.8461317 http://dx.doi.org/10.1109/ICASSP.2018.8461317 ]
Liu Y, Yu H W, Huang B Q, Yue G H and Song B Y. 2021. Blind omnidirectional image quality assessment based on structure and natural features. IEEE Transactions on Instrumentation and Measurement, 70: #5014011 [DOI: 10.1109/TIM.2021.3102691]
Ma K D, Liu W T, Liu T L, Wang Z and Tao D C. 2017. DipIQ: blind image quality assessment by learning-to-rank discriminable image pairs. IEEE Transactions on Image Processing, 26(8): 3951-3964 [DOI: 10.1109/TIP.2017.2708503]
Ma K D, Liu W T, Zhang K, Duanmu Z F, Wang Z and Zuo W M. 2018. End-to-end blind image quality assessment using deep neural networks. IEEE Transactions on Image Processing, 27(3): 1202-1213 [DOI: 10.1109/TIP.2017.2774045]
Madhusudana P C and Soundararajan R. 2019. Subjective and objective quality assessment of stitched images for virtual reality. IEEE Transactions on Image Processing, 28(11): 5620-5635 [DOI: 10.1109/TIP.2019.2921858]
Min X K, Zhai G T, Gu K, Liu Y T and Yang X K. 2018. Blind image quality estimation via distortion aggravation. IEEE Transactions on Broadcasting, 64(2): 508-517 [DOI: 10.1109/TBC.2018.2816783]
Min X K, Zhai G T, Zhou J T, Farias M C Q and Bovik A C. 2020. Study of subjective and objective quality assessment of audio-visual signals. IEEE Transactions on Image Processing, 29: 6054-6068 [DOI: 10.1109/TIP.2020.2988148]
Moorthy A K and Bovik A C. 2010. A two-step framework for constructing blind image quality indices. IEEE Signal Processing Letters, 17(5): 513-516 [DOI: 10.1109/LSP.2010.2043888]
Moorthy A K and Bovik A C. 2011. Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Transactions on Image Processing, 20(12): 3350-3364 [DOI: 10.1109/TIP.2011.2147325]
Nie L, Lin C Y, Liao K, Liu S C and Zhao Y. 2021. Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Transactions on Image Processing, 30: 6184-6197 [DOI: 10.1109/TIP.2021.3092828]
Niu Y Z, Zhong Y N, Guo W Z, Shi Y Q and Chen P K. 2019. 2D and 3D image quality assessment: a survey of metrics and challenges. IEEE Access, 7: 782-801 [DOI: 10.1109/ACCESS.2018.2885818]
Shao F, Fu Z Q, Jiang Q P, Jiang G Y and Ho Y S. 2021. Transformation-aware similarity measurement for image retargeting quality assessment via bidirectional rewarping. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(5): 3053-3067 [DOI: 10.1109/TSMC.2019.2917496]
Sui X J, Ma K D, Yao Y R and Fang Y M. 2021. Perceptual quality assessment of omnidirectional images as moving cam era videos. IEEE Transactions on Visualization and Computer Graphics: #3050888 [ DOI: 10.1109/TVCG.2021.3050888 http://dx.doi.org/10.1109/TVCG.2021.3050888 ]
Sun W, Gu K, Zhai G T, Ma S W, Lin W S and Calle P L. 2017a. CVIQD: subjective quality evaluation of compressed virtual reality images//Proceedings of 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China: IEEE: 3450-3454 [ DOI: 10.1109/ICIP.2017.8296923 http://dx.doi.org/10.1109/ICIP.2017.8296923 ]
Sun W, Min X K, Zhai G T, Gu K, Duan H Y and Ma S W. 2020. MC360IQA: a multi-channel CNN for blind 360-degree image quality assessment. IEEE Journal of Selected Topics in Signal Processing, 14(1): 64-77 [DOI: 10.1109/JSTSP.2019.2955024]
Sun Y L, Lu A and Yu L. 2017b. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE Signal Processing Letters, 24(9): 1408-1412 [DOI: 10.1109/LSP.2017.2720693]
Video Quality Experts Group (VQEG). 2000. Final report from the video quality experts group on the validation of objective models of video quality assessment [EB/OL ] . [2020-07-08 ] . http://videoclarity.com/PDF/COM-80E_final_report.pdf http://videoclarity.com/PDF/COM-80E_final_report.pdf
Wang S Q, Gu K, Ma S W and Gao W. 2016. Joint chroma downsampling and upsampling for screen content image. IEEE Transactions on Circuits and Systems for Video Technology, 26(9): 1595-1609 [DOI: 10.1109/TCSVT.2015.2461891]
Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612 [DOI: 10.1109/TIP.2003.819861]
Wang Z M. 2015. Review of no-reference image quality assessment. Acta Automatica Sinica, 41(6): 1062-1079
王志明. 2015. 无参考图像质量评价综述. 自动化学报, 41(6): 1062-1079 [DOI: 10.16383/j.aas.2015.c140404]
Wu J J. 2014. Image Information Perception and Quality Assessment based on the Human Visual System. Xi′an: Xidian University
吴金建. 2014. 基于人类视觉系统的图像信息感知和图像质量评价. 西安: 西安电子科技大学
Wu J J, Ma J P, Liang F H, Dong W S, Shi G M and Lin W S. 2020. End-to-end blind image quality prediction with cascaded deep neural network. IEEE Transactions on Image Processing, 29: 7414-7426 [DOI: 10.1109/TIP.2020.3002478]
Wu Q B, Li H L, Meng F M and Ngan K N. 2018. Toward a blind quality metric for temporally distorted streaming video. IEEE Transactions on Broadcasting, 64(2): 367-378 [DOI: 10.1109/TBC.2017.2786023]
Xu W and Mulligan J. 2010. Performance evaluation of color correction approaches for automatic multi-view image and video stitching//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, USA: IEEE: 263-270 [ DOI: 10.1109/CVPR.2010.5540202 http://dx.doi.org/10.1109/CVPR.2010.5540202 ]
Xu J H, Zhou W and Chen Z B. 2021. Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks. IEEE Transactions on Circuits and Systems for Video Technology, 31(5): 1724-1737 [DOI: 10.1109/TCSVT.2020.3015186]
Xu M, Li C, Liu Y F, Deng X and Lu J X. 2017. A subjective visual quality assessment method of panoramic videos//Proceedings of 2017 IEEE International Conference on Multimedia and Expo (ICME). Hong Kong, China: IEEE: 517-522 [ DOI: 10.1109/ICME.2017.8019351 http://dx.doi.org/10.1109/ICME.2017.8019351 ]
Yang J C, Liu T L, Jiang B, Song H B and Lu W. 2018. 3D panoramic virtual reality video quality assessment based on 3D convolutional neural networks. IEEE Access, 6: 38669-38682 [DOI: 10.1109/ACCESS.2018.2854922]
Yang X H, Li F and Liu H T. 2019. Asurvey of DNN methods for blind image quality assessment. IEEE Access, 7: 123788-123806 [DOI: 10.1109/ACCESS.2019.2938900]
Yuan Y T, Fang F M and Zhang G X. 2021. Superpixel-based seamless image stitching for UAV images. IEEE Transactions on Geoscience and Remote Sensing, 59(2): 1565-1576 [DOI: 10.1109/TGRS.2020.2999404]
Yu M, Lakshman H and Girod B. 2015. A framework to evaluate omnidirectional video coding schemes//Proceedings of 2015 IEEE International Symposium on Mixed and Augmented Reality. Fukuoka, Japan: IEEE: 31-36 [ DOI: 10.1109/ISMAR.2015.12 http://dx.doi.org/10.1109/ISMAR.2015.12 ]
Yue G H, Hou C P and Zhou T W. 2019. Blind quality assessment of tone-mapped images considering colorfulness, naturalness, and structure. IEEE Transactions on Industrial Electronics, 66(5): 3784-3793 [DOI: 10.1109/TIE.2018.2851984]
Zakharchenko V, Choi K P and Park J H. 2016. Quality metric for spherical panoramic video//Proceedings Volume 9970, Optics and Photonics for Information Processing X. San Diego, USA: SPIE: 1-9 [ DOI: 10.1117/12.2235885 http://dx.doi.org/10.1117/12.2235885 ]
Zhai G T and Min X K. 2020. Perceptual image quality assessment: a survey. Science China Information Sciences, 63(11): #211301 [DOI: 10.1007/s11432-019-2757-1]
Zhan Y B, Zhang R and Wu Q. 2017. A structural variation classification model for image quality assessment. IEEE Transactions on Multimedia, 19(8): 1837-1847 [DOI: 10.1109/TMM.2017.2689923]
Zhang L, Zhang L and Bovik A C. 2015. A feature-enriched completely blind image quality evaluator. IEEE Transactions onImage Processing, 24(8): 2579-2591 [DOI: 10.1109/TIP.2015.2426416]
Zhang W X, Ma K D, Yan J, Deng D X and Wang Z. 2020. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 30(1): 36-47 [DOI: 10.1109/TCSVT.2018.2886771]
Zheng F. 2020. Design and Research on Real-Time AI Reporting of VR E-Sports News. Harbin: Harbin Institute of Technology
郑菲. 2020. 虚拟现实电竞新闻实时智能报道的设计研究. 哈尔滨: 哈尔滨工业大学
Zheng K K. 2017. Research on Stitching Quality Assessment Method of Virtual Reality Panoramic Video. Xiamen: Xiamen University
郑珂珂. 2017. VR全景视频的拼接质量评价方法研究. 厦门: 厦门大学
Zheng X L, Jiang G Y, Yu M and Jiang H. 2020. Segmented spherical projection-based blind omnidirectional image quality assessment. IEEE Access, 8: 31647-31659 [DOI: 10.1109/ACCESS.2020.2972158]
Zhou Y. 2019. Study on the Objective Quality Assessment Metrics of Virtual View Synthesis. Xuzhou: China University of Mining and Technology
周玉. 2019. 面向虚拟视角合成的客观质量评价方法研究. 徐州: 中国矿业大学
Zhou Y, Li L D, Wang S Q, Wu J J, Fang Y M and Gao X B. 2019. No-reference quality assessment for view synthesis using DoG-based edge statistics and texture naturalness. IEEE Transactions on Image Processing, 28(9): 4566-4579 [DOI: 10.1109/TIP.2019.2912463]
Zhou Y, Li L D, Wu J J, Gu K, Dong W S and Shi G M. 2018. Blind quality index for multiply distorted images using biorder structure degradation and nonlocal statistics. IEEE Transactions on Multimedia, 20(11): 3019-3032 [DOI: 10.1109/TMM.2018.2829607]
Zhou Y, Sun Y J, Li L D, Gu K and Fang Y M. 2021. Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network. IEEE Transactions on Circuits and Systems for Video Technology, 32: 1767-1777[DOI: 10.1109/TCSVT.2021.3081162]
Zhou Z, Zhou Y and Xiao J J. 2015. Survey on augmented virtual environment and augmented reality. Scientia Sinica Informationis, 45(2): 157-180
周忠, 周颐, 肖江剑. 2015. 虚拟现实增强技术综述. 中国科学: 信息科学, 45(2): 157-180 [DOI: 10.1360/N112014-00076]
相关文章
相关作者
相关机构
京公网安备11010802024621
