Objective

2

A video captured with a hand-held device (e.g.

cell-phone or tablet computer) often appears remarkably shaky. It has a great negative impact to people's visual perception and is difficult for the following processings of the video. Therefore

video stabilization has always been one of the focuses in the field of image and video processing for a long time. Many of the previous methods focused primarily on the stability of the final results

while paying little attention to the processing delays. These methods spend too much time and they can only stabilize videos after obtaining most of the video frames. These methods are difficult to handle the scenario that requires low latency processing. To solve this problem

a video stabilization method based on the on-the-fly total variation optimization is proposed.

Method

2

The first step is motion estimation. In order to meet the requirements of low latency. The algorithm first gets the feature points by detecting FAST features that are very fast. It is observed that a point is considered as a feature point when the difference of the pixel value between this point and other points in the neighborhood is large enough. Then it tracks the feature points by KLT to the adjacent frame. After that

it calculates the inter frame homography matrix through the feature points. Homography matrix can accurately describe the translation

rotation and other transformations between the video frames. And then we can get the camera path of the shaky video. The second step is motion smoothing. This is usually the time-consuming step of traditional methods. In this step

we should not only remove the shake of the camera path

but also need to try to avoid that the clipping and distortion rate of video is too high. In order to obtain the better effect of video stabilization and meet the requirements of low latency

the on-the-fly total variation minimization method is used to smooth the shaky path so as to obtain a stable path. This method greatly improves the computation speed. And it has a regularization parameter

so we can control the smoothness of the camera path by adjusting it. It can help us get better results. If we want a smoother camera path

we can increase this regularization parameter. However

this may make the optimization path more different from the original path

and results in excessive loss of image information. The visual result is that the video shows larger black edges. When the parameter is reduced

the smoothing effect may not be obvious and the shake may not be removed effectively. So we need to set the parameter according to the size of the camera path. After that

we can compute the motion compensation matrix through the stable camera path and shaky video. The shaky frames are then deformed according to the transformation matrix. Finally

a stable video can be generated.

Result

2

To test the effectiveness of this algorithm

a public dataset containing multiple categories of videos is used. This algorithm is compared with several video stabilization algorithms and commercial softwares that can provide good video stabilization results. Among these methods

there are three offline methods. They are the bundled camera paths method (BCP)

the deformation stabilizer of Adobe After Effects CC 2015 (AE)

and the online video stabilizer provided by YouTube. The two low latency video stabilization methods are video stabilization method based on Kalman filtering and the MeshFlow method. We compare these methods in two ways and implement these methods on the same computer. First

we count the average consumption time per frame of different methods. At the same time

we count the delay time and the number of delay frames in processing the video. Second

we calculate the video distortion rate and cropping rate of different methods. These two rates are derived from the inter-frame affine transformation of videos that are processed by these methods. In addition

we invite 50 non-professional volunteers to make subjective judgments about the stability of these methods. The eventual experimental results show that

different from these offline methods that need to get all or most of the video frames

our algorithm can obtain the stable video with only one frame latency. This result is similar to the Kalman filtering method. Compared with the MeshFlow method

there are about 15% speed improvements. In terms of distortion

this algorithm is only a little worse than the Kalman filtering method

better than any other methods. Similarly

our algorithm is only a little worse than the BCP method in cropping. But it is better than the others. Generally

our algorithm can produce comparable quality with these offline approaches that are recognized generally as the best ones of all the algorithms in stability. This result is similar to the MeshFlow method and almost completely better than the Kalman filtering method.

Conclusion

2

A video stabilization method based on the on-the-fly total variation optimization is proposed

which can get the stable videos with only one frame latency. The results show that the proposed algorithm can generate pleasing results both in delay performance and stability. Compared with the traditional methods

our algorithm is more suitable for the scenario that needs to get the stabilization results with low latency. However

our algorithm estimate inter-frame motion by detecting and matching feature points

it may fail to obtain good stabilization results for those videos that have minor feature points.