浏览全部资源
扫码关注微信
Received：30 December 2020，
Revised：2021-2-24，
Accepted：03 March 2021，
Published：16 April 2022
移动端阅览
目的
2
基于深度学习的动作识别方法识别准确率显著提升，但仍然存在很多挑战和困难。现行方法在一些训练数据大、分类类别多的数据集以及实际应用中鲁棒性较差，而且许多方法使用的模型参数量较大、计算复杂，提高模型准确度和鲁棒性的同时对模型进行轻量化仍然是一个重要的研究方向。为此，提出了一种基于知识蒸馏的轻量化时空图卷积动作识别融合模型。
方法
2
改进最新的时空卷积网络，利用分组卷积等设计参数量较少的时空卷积子模型; 为了训练该模型，选取两个现有的基于全卷积的模型作为教师模型在数据集上训练，在得到训练好的教师模型后，再利用知识蒸馏的方法结合数据增强技术训练参数量较少的时空卷积子模型; 利用线性融合的方法将知识蒸馏训练得到的子模型融合得到最终的融合模型。
结果
2
在广泛使用的NTU RGB + D数据集上与前沿的多种方法进行了比较，在CS(cross-subject)和CV(cross-view)两种评估标准下
本文模型的准确率分别为90.9 %和96.5 %，与教师模型2s-AGCN(two-stream adaptive graph convolutional networks for skeleton-based action)相比，分别提高了2.4 %和1.4 %; 与教师模型DGNN(directed graph neural network)相比，分别提高了1.0 %和0.4 %; 与MS-AAGCN(multi-stream attention-enhanced adaptive graph convolutional neural network)模型相比，分别提高了0.9 %和0.3 %。
结论
2
本文提出的融合模型，综合了知识蒸馏、数据增强技术和模型融合的优点，使动作识别更加准确和鲁棒。
Objective
2
Skeleton-based action recognition
which is an intensively studied field
aims to classify human actions represented by a sequence of selected key points of the human body into action categories. Skeleton-based action recognition has a wide variety of applications
including human-computer interaction
elderly monitoring
and video understanding. Recognition accuracy has improved significantly in recent years due to the development of deep learning. However
few studies have focused on the number of parameters and the robustness of the model. In previous skeleton-based action recognition methods
convolution with a big kernel size has been used to extract spatial and temporal features for the broad receptive field
leading to an increase in model parameters and more complicated calculations. Many previous studies have confirmed that graph convolution has better performance in skeletal data. However
graph convolution operators are designed manually
and their versatility and robustness are insufficient. Therefore
we hope to design a lightweight temporal convolution module that preserves the large receptive field for temporal feature learning. In the spatial dimension
we aimed for better robustness of the spatial convolution module constructed using two kinds of graph convolutions. We will improve the performance of the model with the help of data enhancement technology to increase the diversity of input data and improve the generalization ability of the model for different perspectives. To this end
a distillation training method that can improve the accuracy of lightweight models is used for model training and a multi-stream spatiotemporal graph convolutional ensemble model is constructed to improve the current methods and increase the accuracy of the skeleton-based action recognition.
Method
2
In this study
we propose a skeleton-based multi-stream ensemble model composed of six sub-networks for action recognition. These sub-networks are divided into two types: directed graph convolutional sub neural network (DGCNNet) and adaptive graph convolutional sub neural network (AGCNNet). Each sub-network is constructed with temporal convolution modules
spatial convolution modules
and attention modules. The temporal convolution module in the sub-network is designed with a 2D depth-wise group convolution layer with a convolution kernel size of 9 × 1 and a normal convolution layer with a convolution kernel size of 1 × 1. Two types of graph convolution-directed graph convolution and adaptive graph convolution-are used in the spatial convolution module to extract spatial features and enhance the robustness of the model. Three self-attention modules between the spatial convolution and the temporal convolution modules are applied over the channel dimension
spatial dimension
and temporal dimension of the features to underscore the informative features. We also introduce a cross-modal distillation training method that can be used to train lighter and more accurate student models with trained teacher models and ground truth to train the sub-networks. The distillation training consists of two steps: teacher model training and student model training. The teacher model is trained using training data and its weights are fixed after the training is completed. The student model is trained with the feature vector encoded by the frozen teacher model and the ground-truth labels of training data. Two previous methods
2s-AGCN(two-stream adaptive graph convolutional networks for skeleton-based action) and directed graph neural networks (DGNN)
are used as teacher models to train the DGCNNet sub-network and the AGCNNet sub-network
respectively
according to the type of graph convolution in the spatial convolution module. Cross-entropy loss is used for teacher model training and a combination of mean squared error loss and cross-entropy loss is used as the final loss for student model training. The student model trained using the distillation training method is not only lighter but also more accurate than the corresponding teacher model. In addition to joints
we also take bones and affine-transformation-augmented data as input to train the student model. Finally
a multi-stream spatiotemporal graph convolutional ensemble model is constructed with six lightweight sub-networks
and it has better robustness and higher accuracy. The accuracies of our model for cross-subject benchmark and cross-view benchmark of NTU RGB+D dataset are 90.9 % and 96.5 %
respectively
higher than many other currently best approaches.
Result
2
We compared our model with the other 14 best models thus far on the widely used NTU RGB+D dataset. Our model achieved a 90.9 % cross-subject accuracy and a 96.5 % cross-view accuracy in terms of the benchmark. A comparison of our model with the teacher model
2s-AGCN
indicated that the accuracy increased by 2.4 % and 1.4 %. When compared with another teacher model
DGNN
the accuracy of our model increased by 1.0 % and 0.4 %; and when compared with the base-line method
spatial temporal graph convolution networks (ST-GCN)
the accuracies of our model are 9.4 % and 8.2 % higher
respectively. In addition
extensive experiments indicated the effectiveness of knowledge distillation on this task and we also explored the effects of the different combinations of input modalities on the final accuracy of the model.
Conclusion
2
In this article
we propose a new multi-stream ensemble model that contains six sub-models trained using the distillation training method
and each sub-model is constructed with spatial convolution modules
temporal convolution modules
and attention modules. The results of the experiment indicate that our model outperforms several state-of-the-art skeleton-based action recognition approaches and that the ensemble algorithm can improve the performance.
Cao Z, Hidalgo G, Simon T, Wei S E and Sheikh Y. 2021. OpenPose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1): 172-186 [DOI: 10.1109/TPAMI.2019.2929257]
Cao Z, Simon T, Wei S E and Sheikh Y. 2017. Realtime multi-person 2D pose estimation using part affinity fields//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 1302-1310 [ DOI: 10.1109/CVPR.2017.143 http://dx.doi.org/10.1109/CVPR.2017.143 ]
Climent-Pérez P, Chaaraoui A A, Padilla-López J R and Flórez-Revuelta F. 2012. Optimal joint selection for skeletal data from RGB-D devices using a genetic algorithm//Proceedings of the 11th Mexican International Conference on Artificial Intelligence. San Luis PotosÃ, Mexico: Springer: 163-174 [ DOI: 10.1007/978-3-642-37798-3_15 http://dx.doi.org/10.1007/978-3-642-37798-3_15 ]
Crasto N, Weinzaepfel P, Alahari K and Schmid C. 2019. MARS: motion-augmented RGB stream for action recognition//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 7874-7883 [ DOI: 10.1109/CVPR.2019.00807 http://dx.doi.org/10.1109/CVPR.2019.00807 ]
Du Y, Wang W and Wang L. 2015. Hierarchical recurrent neural network for skeleton based action recognition//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE: 1110-1118 [ DOI: 10.1109/CVPR.2015.7298714 http://dx.doi.org/10.1109/CVPR.2015.7298714 ]
Fankhauser P, Bloesch M, Rodriguez D, Kaestner R, Hutter M and Siegwart R. 2015. Kinect v2 for mobile robot navigation: evaluation and modeling//Proceedings of 2015 International Conference on Advanced Robotics (ICAR). Istanbul, Turkey: IEEE: 388-394 [ DOI: 10.1109/ICAR.2015.7251485 http://dx.doi.org/10.1109/ICAR.2015.7251485 ]
Hinton G, Vinyals O and Dean J. 2015. Distilling the knowledge in a neural network [EB/OL]. [2020-12-30] . http://export.arxiv.org/pdf/1503.02531.pdf http://export.arxiv.org/pdf/1503.02531.pdf
Kim T S and Reiter A. 2017. Interpretable 3D human action analysis with temporal convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, USA: IEEE: 1623-1631 [ DOI: 10.1109/CVPRW.2017.207 http://dx.doi.org/10.1109/CVPRW.2017.207 ]
Li S, Li W Q, Cook C, Zhu C and Gao Y B. 2018. Independently recurrent neural network (IndRNN): building a longer and deeper RNN//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 5457-5466 [ DOI: 10.1109/CVPR.2018.00572 http://dx.doi.org/10.1109/CVPR.2018.00572 ]
Liu H, Tu J and Liu M. 2017a. Two-stream 3D convolutional neural network for skeleton-based action recognition [EB/OL]. [2020-12-30] . http://export.arxiv.org/pdf/1705.08106.pdf http://export.arxiv.org/pdf/1705.08106.pdf
Liu J, Shahroudy A, Xu D and Wang G. 2016. Spatio-temporal LSTM with trust gates for 3D human action recognition//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 816-833 [ DOI: 10.1007/978-3-319-46487-9_50 http://dx.doi.org/10.1007/978-3-319-46487-9_50 ]
Liu M Y, Liu H and Chen C. 2017b. Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognition, 68: 346-362 [DOI: 10.1016/j.patcog.2017.02.030]
Parikh A P, Täckström O, Das D and Uszkoreit J. 2016. A decomposable attention model for natural language inference//Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing. Austin, USA: Association for Computational Linguistics: 2249-2255 [ DOI: 10.18653/v1/D16-1244 http://dx.doi.org/10.18653/v1/D16-1244 ]
Peng W, Hong X P, Chen H Y and Zhao G Y. 2020a. Learning graph convolutional network for skeleton-based human action recognition by neural searching//Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York, USA: AAAI: 2669-2676 [ DOI: 10.1609/aaai.v34i03.5652 http://dx.doi.org/10.1609/aaai.v34i03.5652 ]
Peng W, Shi J G, Xia Z Q and Zhao G Y. 2020b. Mix dimension in poincarégeometry for 3D skeleton-based action recognition//Proceedings of the 28th ACM International Conference on Multimedia. Lisboa, Portugal: ACM: 1432-1440 [ DOI: 10.1145/3394171.3413910 http://dx.doi.org/10.1145/3394171.3413910 ]
Shahroudy A, Liu J, Ng T T and Wang G. 2016. NTU RGB+D: A large scale dataset for 3D human activity analysis//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 1010-1019 [ DOI: 10.1109/CVPR.2016.115 http://dx.doi.org/10.1109/CVPR.2016.115 ]
Shi L, Zhang Y F, Cheng J and Lu H Q. 2019a. Skeleton-based action recognition with directed graph neural networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 7904-7913 [ DOI: 10.1109/CVPR.2019.00810 http://dx.doi.org/10.1109/CVPR.2019.00810 ]
Shi L, Zhang Y F, Cheng J and Lu H Q. 2019b. Two-stream adaptive graph convolutional networks for skeleton-based action recognition//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 12018-12027 [ DOI: 10.1109/CVPR.2019.01230 http://dx.doi.org/10.1109/CVPR.2019.01230 ]
Shi L, Zhang Y F, Cheng J and Lu H Q. 2020. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Transactions on Image Processing, 29: 9532-9545 [DOI: 10.1109/TIP.2020.3028207]
Vapnik V and Izmailov R. 2015. Learning using privileged information: similarity control and knowledge transfer. The Journal of Machine Learning Research, 16(1): 2023-2049
Wang C Y, Wang Y Z and Yuille A L. 2013. An approach to pose-based action recognition//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, USA: IEEE: 915-922 [ DOI: 10.1109/CVPR.2013.123 http://dx.doi.org/10.1109/CVPR.2013.123 ]
Wu C, Wu X J and Kittler J. 2019. Spatial residual layer and dense connection block enhanced spatial temporal graph convolutional network for skeleton-based action recognition//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South): IEEE: 1740-1748 [ DOI: 10.1109/ICCVW.2019.00216 http://dx.doi.org/10.1109/ICCVW.2019.00216 ]
Yan S J, Xiong Y J and Lin D H. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI: 7444-7452
Zhang P F, Lan C L, Xing J L, Zeng W J, Xue J R and Zheng N N. 2017. View adaptive recurrent neural networks for high performance human action recognition from skeleton data//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE: 2136-2145 [ DOI: 10.1109/ICCV.2017.233 http://dx.doi.org/10.1109/ICCV.2017.233 ]
相关文章
相关作者
相关机构
京公网安备11010802024621
