浏览全部资源
扫码关注微信
纸质出版日期： 2022-02-16 ，
录用日期： 2021-07-06
移动端阅览
金康俊, 柴雄力, 邵枫. 时空联合视差优化的立体视频重定向[J]. 中国图象图形学报, 2022,27(2):614-627.
Kangjun Jin, Xiongli Chai, Feng Shao. Optimizing spatiotemporal disparities for stereoscopic video retargeting[J]. Journal of Image and Graphics, 2022,27(2):614-627.
目的
2
智能适配显示的图像/视频重定向技术近年受到广泛关注。与图像重定向以及2D视频重定向相比，3D视频重定向需要同时考虑视差保持和时域保持。现有的3D视频重定向方法虽然考虑了视差保持却忽略了对视差舒适度的调整，针对因视差过大和视差突变造成视觉不舒适度这一问题，提出了一种基于时空联合视差优化的立体视频重定向方法，将视频视差范围控制在舒适区间。
方法
2
在原始视频上建立均匀网格，并提取显著信息和视差，进而得到每个网格的平均显著值；根据相似性变化原理构建形状保持能量项，利用目标轨迹以及原始视频的视差变化构建时域保持能量项，并结合人眼辐辏调节原理构建视差舒适度调整能量项；结合各个网格的显著性，联合求解所有能量项得到优化后的网格顶点坐标，将其用于确定网格形变，从而生成指定宽高比的视频。
结果
2
实验结果表明，与基于细缝裁剪的立体视频重定向方法对比，本文方法在形状保持、时域保持及视差舒适度方面均具有更好的性能。另外，使用现有的客观质量评价方法对重定向结果进行评价，本文方法客观质量评价指标性能优于均匀缩放和细缝裁剪的视频重定向方法，时间复杂度较低，每帧的时间复杂度至少比细缝裁剪方法降低了98%。
结论
2
提出的时空联合的视差优化方法同时在时域和舒适度上对视差进行优化，并考虑了时域保持，具有良好的视差优化与时域保持效果，展现了较高的稳定性和鲁棒性。本文方法能够用于3D视频的重定向，在保持立体视觉舒适性的同时适配不同尺寸的3D显示屏幕。
Objective
2
In recent years
with the rapid development of the digital video photography
people pay more attention to the imaging quality of videos with the increasing demand for the immersive experience. Therefore
it is a meaningful challenge to adjust stereoscopic content to the required size to accommodate the different resolutions of 3D display devices. Stereoscopic video retargeting is quite different from image retargeting and 2D video retargeting aims to minimize the shape distortions and optimize disparities with temporal coherence in resizing a stereoscopic video. More constraints of 3D video retargeting should be considered
such as
temporal coherence and depth information. However
the existing 3D video retargeting methods usually consider minimizing the depth distortion but not take the visual experience. Many studies show that the unsuitable disparity causes the visual discomfort and visual fatigue. As a result
someone does not have a great experience in 3D video watching. To solve this problem
our study proposes a method to control the disparity into a comfortable range after 3D video retargeting. Our method considers disparity remapping with temporal coherence and angular parallax. Furthermore
retargeting video may shaky and flickering because of the incoherence between frames. Mesh motion trajectory is utilized to constrain the retargeting video for minimizing the shaking. Considering the above factors
a stereoscopic video retargeting method via mesh warping in optimizing spatiotemporal disparities is proposed in this paper.
Method
2
First
the mesh in first frame is constructed
and the stereoscopic saliency based on disparity and edge information is estimated. Then
the mean saliency of each grid for mesh importance is computed. Next
the vertex of the mesh in the original video is traced to obtain the vertex trajectories according to the optic flow method. Temporal coherence between the original and retargeted videos can be established by the motion trajectories. Shape distortion is of great importance in affecting the quality of video retargeting. Generally
a high geometric similarity means to a low shape distortion. Therefore
according to significance information
high similarity between the original and deformed grids in those high-significance regions and low similarity in the low-significance regions are maintained. Thus
the shape-preserving energy term is established by using the similarity transformation to minimize the shape distortion. As vergence-accommodation conflict in 3D display may cause fatigue and discomfort
remapping the disparity map into anthropogenic disparity range to reduce the visual fatigue is substantial. After remapping the disparity map
the disparity change with temporal coherence needs to be controlled. In the end
the total energy term is obtained by adding all the energy terms
the optimal grids are obtained by using the linear least square method
and the original video is mapped to obtain the retargeted video.
Result
2
Compared with the existing seam carving based stereoscopic video retargeting method
the proposed method achieves a better performance in terms of shape preservation
temporal coherence preservation and disparity remapping energy terms. The objective performance evaluated using the existing objective assessment methods are also higher than the comparison methods.
Conclusion
2
A spatiotemporal disparity optimization method
which remaps the video disparity to a comfortable range with temporal coherence
is proposed in this paper. A retargeted video that not only satisfies the viewing comfort but also causes less disparity saltation in time domain can be obtained using this method. A stereoscopic video retargeting method based on grid deformation
which according to the saliency information of the video and a grid deformation equation is established to resize the stereoscopic video into our desired resolution
is proposed in this paper. Results show that the proposed method has an excellent performance in shape preserving
time coherence preserving
and disparity optimization. In the next work
combining the cropping method with the grid deformation method to perform stereoscopic video retargeting and reduce the distortion of the salient target further will be considered. The method proposed in this paper can be used in 3D video retargeting with great stereo visual comfort.
立体视频重定向网格形变时空视差优化视频时间一致性立体视觉舒适度立体显著
stereoscopic video retargetingmesh warpingspatio-temporal disparity optimizationtemporal coherencevisual experiencestereoscopic saliency
Avidan S and Shamir A. 2007. Seam carving for content-aware image resizing. ACM Transactions on Graphics, 26(3): #10[DOI:10.1145/1276377.1276390]
Basha T D, Moses Y and Avidan S. 2013. Stereo seam carving a geometrically consistent approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(10): 2513-2525[DOI:10.1109/TPAMI.2013.46]
Chai X L, Shao F, Jiang Q P and Jiang G Y. 2019. Stereoscopic image recomposition based on mesh deformation. Journal of Image and Graphics, 24(3): 334-345
柴雄力, 邵枫, 姜求平, 蒋刚毅. 2019. 基于网格形变的立体图像内容重组. 中国图象图形学报, 24(3): 334-345)[DOI:10.11834/jig.180359]
Chang C H, Liang C K and Chuang Y Y. 2011. Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Transactions on Multimedia, 13(4): 589-601[DOI:10.1109/TMM.2011.2116775]
Chen H R, Wang B, Pan T X, Zhou L W and Zeng H. 2018. CropNet: real-time thumbnailing//Proceedings of the 26th ACM international conference on Multimedia. Seoul, Korea (South): Association for Computing Machinery: 81-89[DOI: 10.1145/3240508.3240517http://dx.doi.org/10.1145/3240508.3240517]
Chen J S, Bai G C, Liang S H and Li Z Q. 2016. Automatic image cropping: a computational complexity study//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 507-515[DOI:10.1109/CVPR.2016.61http://dx.doi.org/10.1109/CVPR.2016.61]
Cormack R and Fox R. 1985. The computation of disparity and depth in stereograms. Perception and Psychophysics, 38(4): 375-380[DOI:10.3758/BF03207166]
Fu Z Q, Shao F, Jiang Q P, Meng X C and Ho Y S. 2020. Subjective and objective quality assessment for stereoscopic 3D image retargeting. IEEE Transactions on Multimedia, 23: 2100-2113[DOI:10.1109/TMM.2020.3008054]
Guo Y W, Liu F, Shi J, Zhou Z H and Gleicher M. 2009. Image retargeting using mesh parametrization. IEEE Transactions on Multimedia, 11(5): 856-867[DOI:10.1109/TMM.2009.2021781]
Jung Y J, Lee S I, Sohn H, Park H W and Ro Y M. 2012. Visual comfort assessment metric based on salient object motion information in stereoscopic video. Journal of Electronic Imaging, 21(1): #011008[DOI:10.1117/1.JEI.21.1.011008]
Kopf S, Guthier B, Hipp C, Kiess J and Effelsberg W. 2014. Warping-based video retargeting for stereoscopic video//Proceedings of 2014 IEEE International Conference on Image Processing (ICIP). Paris, France: IEEE: 2898-2902[DOI:10.1109/ICIP.2014.7025586http://dx.doi.org/10.1109/ICIP.2014.7025586]
Lee K Y, Chung C D and Chuang Y Y. 2012. Scene warping: layer-based stereoscopic image resizing//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA: IEEE: 49-56[DOI:10.1109/CVPR.2012.6247657http://dx.doi.org/10.1109/CVPR.2012.6247657]
Li B, Duan L Y, Lin C W, Huang T J and Gao W. 2015. Depth-preserving warping for stereo image retargeting. IEEE Transactions on Image Processing, 24(9): 2811-2826[DOI:10.1109/TIP.2015.2431441]
Li B, Duan L Y, Wang J Q, Ji R R, Lin C W and Gao W. 2014. Spatiotemporal grid flow for video retargeting. IEEE Transactions on Image Processing, 23(4): 1615-1628[DOI:10.1109/TIP.2014.2305843]
Li B, Lin C W, Liu S, Huang T J, Gao W and Kuo C C J. 2018a. Perceptual temporal incoherence aware stereo video retargeting//Proceedings of the 26thACM international conference on Multimedia. Seoul, Korea (South): Association for Computing Machinery: 1501-1509[DOI:10.1145/3240508.3240682http://dx.doi.org/10.1145/3240508.3240682]
Li B, Lin C W, Shi B X, Huang T J, Gao W and Kuo C C J. 2018b. Depth-aware stereo video retargeting//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE: 6517-6525[DOI:10.1109/CVPR.2018.00682http://dx.doi.org/10.1109/CVPR.2018.00682]
Lin S S, Lin C H, Kuo Y H and Lee T Y. 2016. Consistent volumetric warping using floating boundaries for stereoscopic video retargeting. IEEE Transactions on Circuits and Systems for Video Technology, 26(5): 801-813[DOI:10.1109/TCSVT.2015.2409711]
Lin W C, Shao F, Jiang G Y and Yu M. 2016. 3D video retargeting based on human visual attention. Journal of Optoelectronics·Laser, 27(3): 303-309
林文崇, 邵枫, 蒋刚毅, 郁梅. 2016. 一种基于人眼视觉注意力的三维视频重定向方法. 光电子·激光, 27(3): 303-309)[DOI:10.16136/j.joel.2016.03.0603]
Luo S J, Sun Y T, Shen I C, Chen B Y and Chuang Y Y. 2015. Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Transactions on Visualization and Computer Graphics, 21(1): 56-67[DOI:10.1109/TVCG.2014.2327979]
Mansfield A, Gehler P, van Gool L and Rother C. 2010. Scene carving: scene consistent image retargeting//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece: Springer: 143-156[DOI:10.1007/978-3-642-15549-9_11http://dx.doi.org/10.1007/978-3-642-15549-9_11]
Rubinstein M, Gutierrez D, Sorkine O and Shamir A. 2010. A comparative study of image retargeting. ACM Transactions on Graphics, 29(6): #160[DOI:10.1145/1866158.1866186]
Rubinstein M, Shamir A and Avidan S. 2008. Improved seam carving for video retargeting. ACM Transactions on Graphics, 27(3): 1-9[DOI:10.1145/1360612.1360615]
Terzić K and Hansard M. 2016. Methods for reducing visual discomfort in stereoscopic 3D: a review. Signal Processing: Image Communication, 47: 402-416[DOI:10.1016/j.image.2016.08.002]
Urvoy M, Barkowsky M and Le Callet P. 2013. How visual fatigue and discomfort impact 3D-TV quality of experience: a comprehensive review of technological, psychophysical, and psychological factors. Annals of Telecommunications-Annales Des Télécommunications, 68(11/12): 641-655[DOI:10.1007/s12243-013-0394-3]
Utsugi K, Shibahara T, Koike K, Takahashi K and Naemura T. 2010. Seam carving for stereo images//Proceedings of 2010 3DTV-Conference: the True Vision-Capture, Transmission and Display of 3D Video. Tampere, Finland: IEEE: 1-4[DOI:10.1109/3DTV.2010.5506316http://dx.doi.org/10.1109/3DTV.2010.5506316]
Wang W G, Shen J B, Yu Y Z and Ma K L. 2017. Stereoscopic thumbnail creation via efficient stereo saliency detection. IEEE Transactions on Visualization and Computer Graphics, 23(8): 2014-2027[DOI:10.1109/TVCG.2016.2600594]
Wang Y S, Fu H B, Sorkine O, Lee T Y and Seidel H P. 2009. Motion-aware temporal coherence for video resizing. ACM Transactions on Graphics, 28(5): 1-10[DOI:10.1145/1618452.1618473]
Wolf L, Guttmann M and Cohen-Or D. 2007. Non-homogeneous content-driven video-retargeting//Proceedings of the 11th IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil: IEEE: 1-6[DOI:10.1109/ICCV.2007.4409010http://dx.doi.org/10.1109/ICCV.2007.4409010]
Yan B, Sun K R and Liu L. 2013. Matching-area-based seam carving for video retargeting. IEEE Transactions on Circuits and Systems for Video Technology, 23(2): 302-310[DOI:10.1109/TCSVT.2012.2203740]
Yen T C, Tsai C M and Lin C W. 2011. Maintaining temporal coherence in video retargeting using mosaic-guided scaling. IEEE Transactions on Image Processing, 20(8): 2339-2351[DOI:10.1109/TIP.2011.2114357]
Zhang G X, Cheng M M, Hu S M and Martin R R. 2009. A shape-preserving approach to image resizing. Computer Graphics Forum, 28(7): 1897-1906[DOI:10.1111/j.1467-8659.2009.01568.x]
Zhang Y B, Lin W S, Zhang X F, Fang Y M and Li L D. 2016. Aspect Ratio Similarity (ARS) for image retargeting quality assessment//Proceedings of 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Shanghai, China: IEEE: 1080-1084[DOI:10.1109/ICASSP.2016.7471842http://dx.doi.org/10.1109/ICASSP.2016.7471842]
相关文章
相关作者
相关机构
京公网安备11010802024621