浏览全部资源
扫码关注微信
Received：26 August 2020，
Revised：2020-10-21，
Accepted：28 October 2020，
Published：16 January 2021
移动端阅览
目的
2
针对视觉目标跟踪（video object tracking，VOT）和视频对象分割（video object segmentation，VOS）问题，研究人员提出了多个多任务处理框架，但是该类框架的精确度和鲁棒性较差。针对此问题，本文提出一个融合多尺度上下文信息和视频帧间信息的实时视觉目标跟踪与视频对象分割多任务的端到端框架。
方法
2
文中提出的架构使用了由空洞深度可分离卷积组成的更加多尺度的空洞空间金字塔池化模块，以及具备帧间信息的帧间掩模传播模块，使得网络对多尺度目标对象分割能力更强，同时具备更好的鲁棒性。
结果
2
本文方法在视觉目标跟踪VOT-2016和VOT-2018数据集上的期望平均重叠率（expected average overlap，EAO）分别达到了0.462和0.408，分别比SiamMask高了0.029和0.028，达到了最先进的结果，并且表现出更好的鲁棒性。在视频对象分割DAVIS（densely annotated video segmentation）-2016和DAVIS-2017数据集上也取得了有竞争力的结果。其中，在多目标对象分割DAVIS-2017数据集上，本文方法比SiamMask有更好的性能表现，区域相似度的杰卡德系数的平均值
J
M
和轮廓精确度的F度量的平均值
F
M
分别达到了56.0和59.0，并且区域和轮廓的衰变值
J
D
和
F
D
都比SiamMask中的低，分别为17.9和19.8。同时运行速度为45帧/s，达到了实时的运行速度。
结论
2
文中提出的融合多尺度上下文信息和视频帧间信息的实时视觉目标跟踪与视频对象分割多任务的端到端框架，充分捕捉了多尺度上下文信息并且利用了视频帧间的信息，使得网络对多尺度目标对象分割能力更强的同时具备更好的鲁棒性。
Objective
2
Visual object tracking (VOT) is widely used in scenes
such as car navigation
automatic video surveillance
and human-computer interaction. It is a basic research task in video applications and needs to infer the correspondence between the target and the frame. Given the position of any object of interest in the first frame of the video
its position is estimated in all subsequent frames with the highest possible accuracy. Similar to VOT
semi-supervised video object segmentation (VOS) requires segmentation of target objects on subsequent video sequences given the initial frame mask. It is also a basic research task of computer vision. However
the target object may experience large changes in pose
proportion
and appearance in the entire video sequence. It may encounter abnormal conditions
such as occlusion
rapid movement
and truncation. Therefore
performing robust VOT and VOS in a semi-supervised manner in video sequences is challenging. The continuous nature of the video sequence itself brings additional contextual information to VOS. The interframe consistency of video enables the network to effectively transfer information from frame to frame. In VOS
the information from previous frames can be regarded as temporal context and can provide useful hints for subsequent predictions. Therefore
the effective use of additional information brought by video is extremely important for video tasks. For the research of VOT and VOS
various multitask processing frameworks have been proposed by scholars. However
the accuracy and robustness of such frameworks are poor. This paper proposes a multitask end-to-end framework for real-time VOT and VOS to address these problems. This framework combines multi-scale context information and video interframe information.
Method
2
In this work
depthwise convolution is changed from depthwise convolution to atrous depthwise convolution
thereby forming the atrous depthwise separable convolution. In accordance with different atrous ratios
the convolution can have different receptive fields while maintaining its lightweight. This study designs an atrous spatial pyramid pooling module with many atrous ratios composed of atrous depthwise separable convolution and applies it to the VOS branch. The network can capture multi-scale context. This work uses 1
3
6
9
12
24
36
and 48 atrous ratios to convolve the feature map with different receptive fields and utilizes adaptive pooling for the feature map. These feature maps are concatenated
and a convolution kernel of 1×1 is used to transform the feature map channel. The feature map outputted by the model has rich multi-scale context information through these operations. This module uses the atrous depthwise separable convolution with different atrous rates for enabling the network to predict multi-scale targets. Continuity is a unique property of video sequences and causes additional contextual information to video tasks. The interframe consistency of video enables the network to effectively transfer information from frame to frame. In the VOS
the information from previous frames can be regarded as temporal context information and can provide useful hints for subsequent predictions. Therefore
the effective use of additional information brought by video is extremely important for video tasks. Inspired by the reference-guided mask propagation algorithm
a mask propagation module is added to the VOS branch for providing location and segmentation information to the network. The proposed mask propagation module is composed of 3×3 convolutions with atrous ratios of 2
3
and 6. In our architecture
a multi-scale atrous spatial pyramid pooling module composed of atrous depthwise separable convolutions and an interframe mask propagation module with interframe information are used. These modules provide the network with strong ability to segment multi-scale target objects and has better robustness.
Result
2
All experiments in this work are performed using NVIDIA TITAN X graphics cards. The network in this article is trained in two stages. The training sets used in different stages are different due to their different nature. In the first stage of training
this work uses Youtube-VOS
common objects in context(COCO)
DETection(ImageNet-DET)
and ImageNet-VID (VIDeo) datasets. For the datasets without mask ground truth
the mask branch is not trained. For a video sequence with only a single frame
the picture and mask of the previous frame are set in the interframe mask propagation module to be the same as the current frame. Inspired by SiamMask
this article uses stochastic gradient descent optimizer algorithm and a warm-up training strategy. The learning rate increases from 1×10
-3
to 5×10
-3
in the first 5 epochs. A logarithmic decay strategy was then used to reduce the learning rate to 2.5×10
-4
through 15 epochs of learning. In the second stage
this article only uses the Youtube-VOS and COCO datasets for training. The two datasets have mask truth values to improve the segmentation effect of video objects. The second stage uses a logarithmic decay strategy to reduce the learning rate from 2.5×10
-4
to 1.0×10
-4
through 20 epochs. The expected average overlaps of the proposed method on the VOT-2016 and VOT-2018 datasets reach 0.462 and 0.408
respectively
which is approximately 0.03 higher than SiamMask. The proposed method achieves advanced results and shows better robustness. Competitive results are also achieved on the DAVIS-2016 and DAVIS-2017 datasets of VOS. On DAVIS-2017 dataset of multitarget object segmenta
tion
the proposed method has better performance than SiamMask. The evaluation indexes
J
M
and
F
M
reach 56.0 and 59.0
respectively
and the decay values of the region and the contour are
J
D
and
F
D
. Their values are 17.9 and 19.8
respectively
which are lower than those in SiamMask. The running speed is 45 frames per second
reaching a real-time running speed.
Conclusion
2
In this study
we proposed a multitask end-to-end framework of real-time VOT and VOS. The proposed method integrates multi-scale context information and video interframe information
fully captures multi-scale context information
and utilizes the information between video frames. These features make the network robust to segmentation of multi-scale target objects.
Bertinetto L, Valmadre J, Henriques J F, Vedaldi A and Torr P H S. 2016. Fully-convolutional siamese networks for object tr acking//Proceedings of European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 850-865[ DOI:10.1007/978-3-319-48881-3_56 http://dx.doi.org/10.1007/978-3-319-48881-3_56 ]
Bolme D S, Beveridge J R, Draper B A and Lui Y M. 2010. Visual object tracking using adaptive correlation filters//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, USA: IEEE: 2544-2550[ DOI:10.1109/CVPR.2010.5539960 http://dx.doi.org/10.1109/CVPR.2010.5539960 ]
Caelles S, Maninis K K, Pont-Tuset J, Leal-TaixéL, Cremers D and van Gool L. 2017. One-shot video object segmentation//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5320-5329[ DOI:10.1109/CVPR.2017.565 http://dx.doi.org/10.1109/CVPR.2017.565 ]
Chen L C, Zhu Y K, Papandreou G, Schroff F and Adam H. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 833-851[ DOI:10.1007/978-3-030-01234-2_49 http://dx.doi.org/10.1007/978-3-030-01234-2_49 ]
Cheng J C, Tsai Y H, Hung W C, Wang S J and Yang M H. 2018. Fast and accu rate online video object segmentation via tracking parts//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7415-7424[ DOI:10.1109/CVPR.2018.00774 http://dx.doi.org/10.1109/CVPR.2018.00774 ]
Danelljan M, Häger G, Khan F S and Felsberg M. 2017. Discriminative scale space tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8):1561-1575[DOI:10.1109/TPAMI.2016.2609928]
Guo D Y, Wang J, Cui Y, Wang Z H and Chen S Y. 2020. SiamCAR: siamese fully convolutional classification and regression for visual tracking//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 6268-6276[ DOI:10.1109/CVPR42600.2020.00630 http://dx.doi.org/10.1109/CVPR42600.2020.00630 ]
Guo Q, Feng W, Zhou C, Huang R, Wan L and Wang S. 2017. Learning dynamic siamese network for visual object tracking//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 1781-1789[ DOI:10.1109/ICCV.2017.196 http://dx.doi.org/10.1109/ICCV.2017.196 ]
He A F, Luo C, Tian X M and Zeng W J. 2018. A twofold siamese network for real-time object tracking//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognitio n. Salt Lake City, USA: IEEE: 4834-4843[ DOI:10.1109/CVPR.2018.00508 http://dx.doi.org/10.1109/CVPR.2018.00508 ]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI:10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Henriques J F, Caseiro R, Martins P and Batista J. 2015. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):583-596[DOI:10.1109/TPAMI.2014.2345390]
Johnander J, Danelljan M, Brissman E, Khan F S and Felsberg M. 2019. A generative appearance model for end-to-end video object segmentation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 8945-8954[ DOI:10.1109/CVPR.2019.00916 http://dx.doi.org/10.1109/CVPR.2019.00916 ]
Li B, Wu W, Wang Q, Zhang F Y, Xing J L and Yan J J. 2019. Siamrpn++: evolution of siamese visual tracking with very deep networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 4277-4286[ DOI:10.1109/CVPR.2019.00441 http://dx.doi.org/10.1109/CVPR.2019.00441 ]
Li B, Yan J J, Wu W, Zhu Z and Hu X L. 2018. High performance visual tracking with siamese region proposal network//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 8971-8980[ DOI:10.1109/CVPR.2018.00935 http://dx.doi.org/10.1109/CVPR.2018.00935 ]
Li F, Yao Y J, Li P H, Zhang D, Zuo W M and Yang M H. 2017. Integrating boundary and center correlation filters for visual tracking with aspect ratio variation//Proceedings of 2017 IEEE International Conference on Computer Vision Workshops. Venice, Italy: IEEE: 2001-2009[ DOI:10.1109/ICCVW.2017.234 http://dx.doi.org/10.1109/ICCVW.2017.234 ]
Li X, Zha Y F, Zhang T Z, Cui Z, Zuo W M, Hou Z Q, Lu H C and Wang H Z. 2019. Survey of visual object tracking algorithms based on deep learning. Journal of Image and Graphics, 24(12):2057-2080
李玺, 查宇飞, 张天柱, 崔振, 左旺孟, 侯志强, 卢湖川, 王菡子. 2019.深度学习的目标跟踪算法综述.中国图象图形学报, 24(12):2057-2080)[DOI:10.11834/jig.190372]
Maninis K K, Caelles S, Chen Y, Pont-Tuset J, Leal-Taixé L, Cremers D and van Gool L. 2019. Video object segmentation without temporal information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(6):1515-1530[DOI:10.1109/TPAMI.2018.2838670]
Oh S W, Lee J Y, Sunkavalli K and Kim S J. 2018. Fast video object segmentation by reference-guided mask propagation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7376-7385[ DOI:10.1109/CVPR.2018.00770 http://dx.doi.org/10.1109/CVPR.2018.00770 ]
Perazzi F, Khoreva A, Benenson R, Schiele B and Sorkine-Hornung A. 2017. Learning video object segmentation from static images//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 3491-3500[ DOI:10.1109/CVPR.2017.372 http://dx.doi.org/10.1109/CVPR.2017.372 ]
Perazzi F, Pont-Tuset J, McWilliams B, van Gool L, Gross M and Sorkine-Hornung A. 2016. A benchmark dataset and evaluation methodology for video object segmentation//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 724-732[ DOI:10.1109/CVPR.2016.85 http://dx.doi.org/10.1109/CVPR.2016.85 ]
Ren S Q, He K M, Girshick R and Sun J. 2017. Faster R-CNN:towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-1149[DOI:10.1109/TPAMI.2016.2577031]
Smeulders A W M, Chu D M, Cucchiara R, Calderara S, Dehghan A and Shah M. 2014. Visual tracking:an experimental survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1442-1468[DOI:10.1109/TPAMI.2013.230]
Valmadre J, Bertinetto L, Henriques J, Vedaldi A and Torr P H S. 2017. End-to-end representation learning for correlation filter based tracking//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5000-5008[ DOI:10.1109/CVPR.2017.531 http://dx.doi.org/10.1109/CVPR.2017.531 ]
Voigtlaender P and Leibe B. 2017. Online adaptation of convolutional neural networks for video object segmentation[EB/OL ] .[2020-08-26 ] . https://arxiv.org/pdf/1706.09364.pdf https://arxiv.org/pdf/1706.09364.pdf
Voigtlaender P, Chai Y N, Schroff F, Adam H, Leibe B and Chen L C. 2019. FEELVOS: fast end-to-end embedding learning for video object segmentation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 9473-9482[ DOI:10.1109/CVPR.2019.00971 http://dx.doi.org/10.1109/CVPR.2019.00971 ]
Wang Q, Z hang L, Bertinetto L, Hu W M and Torr P H S. 2019. Fast online object tracking and segmentation: a unifying approach//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 1328-1338[ DOI:10.1109/CVPR.2019.00142 http://dx.doi.org/10.1109/CVPR.2019.00142 ]
Yang L, Wang Y, Xiong X, Yang J and Katsaggelos A K. 2018. Efficient video object segmentation via network modulation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 6499-6507[ DOI:10.1109/CVPR.2018.00680 http://dx.doi.org/10.1109/CVPR.2018.00680 ]
Zhang L, Varadarajan J, Suganthan P N, Ahuja N and Moulin P. 2017. Robust visual tracking using oblique random forests//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 5825-5834[ DOI:10.1109/CVPR.2017.617 http://dx.doi.org/10.1109/CVPR.2017.617 ]
Zhang Y L, Qian X Y, Zhang M and Ge H J. 2020. Correlation filter target tracking algorithm based on adaptive multifeature fusion. Journal of Image and Graphics, 25(6):1160-1170
张艳琳, 钱小燕, 张淼, 葛红娟. 2020.自适应多特征融合相关滤波目标跟踪.中国图象图形学报, 25(6):1160-1170)[DOI:10.11834/jig.190304]
Zh u Z, Wang Q, Li B, Wu W, Yan J J and Hu W M. 2018. Distractor-aware siamese networks for visual object tracking//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 103-119[ DOI:10.1007/978-3-030-01240-3_7 http://dx.doi.org/10.1007/978-3-030-01240-3_7 ]
相关作者
相关机构
京公网安备11010802024621
