浏览全部资源
扫码关注微信
Received：09 July 2017，
Revised：2017-9-28，
Published：16 February 2018
移动端阅览
目的
2
传统的视频稳像方法为了获得理想的稳像效果，一般耗费较多的计算时间，且存在较长的延时。针对此问题，提出一种即时全变差优化的低延时视频稳像方法。
方法
2
首先利用特征点检测和匹配计算帧间单应变换，得到抖动视频的运动路径；然后通过即时全变差优化方法对抖动路径进行平滑优化，获得稳定的运动路径；最后通过运动补偿，生成稳定的视频。
结果
2
对公共视频数据集中的抖动视频进行稳像效果测试，并与当前稳像效果较好的几种稳像算法和商业软件进行效果和时间对比。在时间方面，统计了不同方法的每帧平均消耗时间和处理延迟帧数，不同于后期处理方法需要得到大部分视频帧才能够进行计算，本文算法能够在只有一帧延时的情况下获得最终的稳像结果，相比于MeshFlow方法有15%左右的速度提升；在稳像效果方面，计算了不同方法稳像后的视频扭曲率和裁剪率，并邀请非专业用户进行了稳定程度的主观判断，本文算法的实验结果并不输于目前被公认较好的3种后期稳像方法，优于Kalman滤波方法。
结论
2
本文所提稳像方法能够兼顾速度和有效性，相对于传统方法，更适合低延时要求的应用场景。
Objective
2
A video captured with a hand-held device (e.g.
cell-phone or tablet computer) often appears remarkably shaky. It has a great negative impact to people's visual perception and is difficult for the following processings of the video. Therefore
video stabilization has always been one of the focuses in the field of image and video processing for a long time. Many of the previous methods focused primarily on the stability of the final results
while paying little attention to the processing delays. These methods spend too much time and they can only stabilize videos after obtaining most of the video frames. These methods are difficult to handle the scenario that requires low latency processing. To solve this problem
a video stabilization method based on the on-the-fly total variation optimization is proposed.
Method
2
The first step is motion estimation. In order to meet the requirements of low latency. The algorithm first gets the feature points by detecting FAST features that are very fast. It is observed that a point is considered as a feature point when the difference of the pixel value between this point and other points in the neighborhood is large enough. Then it tracks the feature points by KLT to the adjacent frame. After that
it calculates the inter frame homography matrix through the feature points. Homography matrix can accurately describe the translation
rotation and other transformations between the video frames. And then we can get the camera path of the shaky video. The second step is motion smoothing. This is usually the time-consuming step of traditional methods. In this step
we should not only remove the shake of the camera path
but also need to try to avoid that the clipping and distortion rate of video is too high. In order to obtain the better effect of video stabilization and meet the requirements of low latency
the on-the-fly total variation minimization method is used to smooth the shaky path so as to obtain a stable path. This method greatly improves the computation speed. And it has a regularization parameter
so we can control the smoothness of the camera path by adjusting it. It can help us get better results. If we want a smoother camera path
we can increase this regularization parameter. However
this may make the optimization path more different from the original path
and results in excessive loss of image information. The visual result is that the video shows larger black edges. When the parameter is reduced
the smoothing effect may not be obvious and the shake may not be removed effectively. So we need to set the parameter according to the size of the camera path. After that
we can compute the motion compensation matrix through the stable camera path and shaky video. The shaky frames are then deformed according to the transformation matrix. Finally
a stable video can be generated.
Result
2
To test the effectiveness of this algorithm
a public dataset containing multiple categories of videos is used. This algorithm is compared with several video stabilization algorithms and commercial softwares that can provide good video stabilization results. Among these methods
there are three offline methods. They are the bundled camera paths method (BCP)
the deformation stabilizer of Adobe After Effects CC 2015 (AE)
and the online video stabilizer provided by YouTube. The two low latency video stabilization methods are video stabilization method based on Kalman filtering and the MeshFlow method. We compare these methods in two ways and implement these methods on the same computer. First
we count the average consumption time per frame of different methods. At the same time
we count the delay time and the number of delay frames in processing the video. Second
we calculate the video distortion rate and cropping rate of different methods. These two rates are derived from the inter-frame affine transformation of videos that are processed by these methods. In addition
we invite 50 non-professional volunteers to make subjective judgments about the stability of these methods. The eventual experimental results show that
different from these offline methods that need to get all or most of the video frames
our algorithm can obtain the stable video with only one frame latency. This result is similar to the Kalman filtering method. Compared with the MeshFlow method
there are about 15% speed improvements. In terms of distortion
this algorithm is only a little worse than the Kalman filtering method
better than any other methods. Similarly
our algorithm is only a little worse than the BCP method in cropping. But it is better than the others. Generally
our algorithm can produce comparable quality with these offline approaches that are recognized generally as the best ones of all the algorithms in stability. This result is similar to the MeshFlow method and almost completely better than the Kalman filtering method.
Conclusion
2
A video stabilization method based on the on-the-fly total variation optimization is proposed
which can get the stable videos with only one frame latency. The results show that the proposed algorithm can generate pleasing results both in delay performance and stability. Compared with the traditional methods
our algorithm is more suitable for the scenario that needs to get the stabilization results with low latency. However
our algorithm estimate inter-frame motion by detecting and matching feature points
it may fail to obtain good stabilization results for those videos that have minor feature points.
Zhang L, Chen X Q, Kong X Y, et al. Geodesic video stabilization in transformation space[J]. IEEE Transactions on Image Processing, 2017, 26(5):2219-2229.[DOI:10.1109/TIP.2017.2676354]
Grundmann M, Kwatra V, Essa I. Auto-directed video stabilization with robust L1 optimal camera paths[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI: IEEE, 2011: 225-232. [ DOI:10.1109/cvpr.2011.5995525 http://dx.doi.org/10.1109/cvpr.2011.5995525 ]
Liu F, Gleicher M, Wang J, et al. Subspace video stabilization[J]. ACM Transactions on Graphics, 2011, 30(1):#4.[DOI:10.1145/1899404.1899408]
Liu S C, Yuan L, Tan P, et al. Bundled camera paths for video stabilization[J]. ACM Transactions on Graphics, 2013, 32(4):#78.[DOI:10.1145/2461912.2461995]
Xing H, Yan J L, Zhang S J. Digital image stabilization using kalman filtering[J]. Acta Armamentarii, 2007, 28(2):175-177.
邢慧, 颜景龙, 张树江.基于Kalman滤波的稳像技术研究[J].兵工学报, 2007, 28(2):175-177.][DOI:10.3321/j.issn:1000-1093.2007.02.011]
Yang J L, Schonfeld D, Mohamed M. Robust video stabilization based on particle filter tracking of projected camera motion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(7):945-954.[DOI:10.1109/TCSVT.2009.2020252]
Zhu Z W, He K, Wang X L. Real-time electronic image stabilizing approach based on CUDA and kalman predictor[J]. Journal of Jilin University:Information Science Edition, 2015, 33(1):45-51.
朱振伍, 何凯, 王新磊.基于CUDA和卡尔曼预测的实时电子稳像方法[J].吉林大学学报:信息科学版, 2015, 33(1):45-51.][DOI:10.3969/j.issn.1671-5896.2015.01.008]
Wang C T, Kim J H, Byun K Y, et al. Robust digital image stabilization using the Kalman filter[J]. IEEE Transactions on Consumer Electronics, 2009, 55(1):6-14.[DOI:10.1109/TCE.2009.4814407]
Liu S C, Tan P, Yuan L, et al. MeshFlow: minimum latency online video stabilization[C]//Computer Vision-ECCV 2016. Cham: Springer, 2016. [ DOI:10.1007/978-3-319-46466-4_48 http://dx.doi.org/10.1007/978-3-319-46466-4_48 ]
Bronshtein I N, Semendyayev K A. Handbook of Mathematics[M]. 3rd ed. New York:Springer-Verlag, 1997.
Rudin L I, Osher S, Fatemi E. Nonlinear total variation based noise removal algorithms[J]. Physica D:Nonlinear Phenomena, 1992, 60(1-4):259-268.[DOI:10.1016/0167-2789(92)90242-F]
Guo C Z, Qin Z Y, Shi W J. TV image denoising model based on energy functionals and HVS[J]. Journal of Image and Graphics, 2014, 19(9):1282-1287.
郭从洲, 秦志远, 时文俊.基于能量泛函和视觉特性的全变分图像降噪模型[J].中国图象图形学报, 2014, 19(9):1282-1287.][DOI:10.11834/jig.20140904]
Frecon J, Pustelnik N, Abry P, et al. On-the-fly approximation of multivariate total variation minimization[J]. IEEE Transactions on Signal Processing, 2016, 64(9):2355-2364.[DOI:10.1109/TSP.2016.2516962]
Zhang L, Xu Q K, Huang H. A Global Approach to Fast Video Stabilization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(2):225-235.[DOI:10.1109/TCSVT.2015.2501941]
Rosten E, Drummond T. Machine learning for high-speed corner detection[C]//Computer Vision-ECCV 2006. Berlin, Heidelberg: Springer, 2006: 430-443. [ DOI:10.1007/11744023_34 http://dx.doi.org/10.1007/11744023_34 ]
Shi J B, Tomasi. Good features to track[C]//Proceedings of 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Seattle, WA: IEEE, 1993. [ DOI:10.1109/CVPR.1994.323794 http://dx.doi.org/10.1109/CVPR.1994.323794 ]
Wang Z Q, Zhang L, Huang H. Multiplane video stabilization[J]. Computer Graphics Forum, 2013, 32(7):265-273.[DOI:10.1111/cgf.12234]
Wahlberg B, Boyd S, Annergren M, et al. An ADMM algorithm for a class of total variation regularized estimation problems[J]. IFAC Proceedings Volumes, 2012, 45(16):83-88.[DOI:10.3182/20120711-3-BE-2027.00310]
Steidl G, Didas S, Neumann J. Splines in higher order TV regularization[J]. International Journal of Computer Vision, 2006, 70(3):241-255.[DOI:10.1007/s11263-006-8066-7]
相关作者
相关机构
京公网安备11010802024621
