浏览全部资源
扫码关注微信
Received：31 May 2023，
Revised：2023-07-31，
Published：16 May 2024
移动端阅览
目的
2
为了更好地实现轻量化的人体姿态估计，在轻量级模型极为有限的资源下实现更高的检测性能。基于高分辨率网络（high resolution network，HRNet）提出了结合密集连接网络的轻量级高分辨率人体姿态估计网络（lightweight high-resolution human estimation combined with densely connected network，LDHNet）。
方法
2
通过重新设计HRNet中的阶段分支结构以及提出新的轻量级特征提取模块，构建了轻量高效的特征提取单元，同时对多分支之间特征融合部分进行了轻量化改进，进一步降低模型的复杂度，最终大幅降低了模型的参数量与计算量，实现了轻量化的设计目标，并且保证了模型的性能。
结果
2
实验表明，在MPII（Max Planck Institute for Informatics）测试集上相比于自顶向下的轻量级人体姿态估计模型LiteHRNet，LDHNet仅通过增加少量参数量与计算量，平均预测准确度即提升了1.5%，与LiteHRNet的改进型DiteHRNet相比也提升了0.9%，在COCO（common objects in context）验证集上的结果表明，与LiteHRNet相比，LDHNet的平均检测准确度提升了3.4%，与DiteHRNet相比也提升了2.3%，与融合Transformer的HRFormer相比，LDHNet在参数量和计算量都更低的条件下有近似的检测性能，在面对实际场景时LDHNet也有着稳定的表现，在同样的环境下LDHNet的推理速度要高于基线HRNet以及LiteHRNet等。
结论
2
该模型有效实现了轻量化并保证了预测性能。
Objective
2
Human pose estimation is a technology that can be widely used in life. In recent years， many excellent high-precision methods have been proposed， but they are often accompanied by a very large model scale， which will encounter the problem of computing power bottleneck in application. Whether for model training or deployment， large models require considerable computing power as the basis. Most of them have low computing power. Similarly， for the scenes in daily life， the equipment needs further applicability and detection speed of the model， which is difficult to achieve by large models. Given such requirements， lightweight human pose estimation has become a hot research field. The main problem is how to achieve high detection accuracy and fast detection speed under the extremely limited number of resources. Lightweight models will inevitably fall into a disadvantage in detection accuracy compared with large models. However， fortunately， from many studies in recent years， the lightweight model can also achieve higher detection accuracy than large ones. A good balance can be reached between them.
Method
2
Based on a high-resolution network （HRNet）， a lightweight high-resolution human pose estimation network combined with a dense connection network （LDHNet） was proposed. First， dense connection and multi-scale were integrated to construct a lightweight and efficient feature extraction unit by redesigning the stage branch structure in HRNet and proposing a new lightweight feature extraction module. Then， the feature extraction module is composed of modules similar to the pyramid structure， and the dilated convolution of three scales is used to obtain a wide range of feature information in the feature map by stacking the multi-layer feature extraction modules and fusing the output of each layer. The concatenation of the output feature map of the feature extraction module is to reuse the feature map and fully extract the information contained in the feature map. These two points can make up for the problem of insufficient utilization of feature information that may exist in the lightweight model and use limited resources to achieve high feature extraction performance. Second， a wide range of cross-branch feature information interactions exists in the original HRNet structure， including feature fusion and the generation of new branches in each stage. LDHNet replaces the convolution in this process with the depthwise separable convolution by changing the size of the feature map through convolution downsampling or upsampling to add with other branch feature maps for feature fusion. This case further reduces the number of parameters of the model based on almost no loss of detection performance. In addition， LDHNet improves the original data preprocessing module and uses the double-branch form to fully extract the information from the original image. Experiments show that considerable information from the original image is of great help to improve the detection performance of the model. LDHNet also uses coordinate attention to reinforce spatial location information.
Result
2
After the above improvements， the size of the model has been greatly reduced to less than one-tenth of the original HRNet. Although some gap still exists between the model size and the current smallest lightweight model LiteHRNet， the design of the lightweight model is not only concerned with the size of the model. This study mainly compares LDHNet with mainstream lightweight models. Through experimental verification on two mainstream datasets of human pose estimation MPII and common objects in context（COCO） dataset and comparison with the current mainstream methods， the following conclusions can be obtained. Compared with the top-down lightweight human pose estimation LiteHRNet on the MPII test set， the average prediction accuracy of LDHNet is improved by 1.5% by only adding a small number of parameters and calculations. The results on the COCO validation set show that compared with LiteHRNet， the average detection accuracy of LDHNet is improved by 3.4%. Compared with the improved DiteHRNet of LiteHRNet， the detection accuracy of LDHNet is improved by 2.3%. Compared with the HRFormer fused with Transformer， the detection accuracy of LDHNet is the same when the scale is smaller. The experimental results on public datasets show that LDHNet achieves excellent results in model lightweight. LDHNet achieves a very good balance between lightweight and model detection performance. For lightweight human pose estimation， the performance of LDHNet is similar to that of Transformer. In addition to experimental verification in the public data set， this study also tests the inference speed and detection accuracy of the model in the actual scene. Compared with the original HRNet， LDHNet has a significant improvement in the detection speed under GPU acceleration. Compared with other lightweight methods， such as LiteHRNet， LDHNet can make full use of hardware resources. The results of the actual test show that LDHNet has a stable performance in the face of the actual scene. The above experimental results show that LDHNet has achieved the design expectations. Moreover， using extremely limited resources， compared with other lightweight human pose estimation models， LDHNet can make full use of all the computing power regardless of the level of hardware computing power. The detection accuracy is greatly improved， and the inference speed of the model is also significantly improved.
Conclusion
2
For LDHNet， some problems also need to be solved， mainly in that the reasoning speed of the model has not reached the level matching the improvement of the model in terms of lightweight. Follow-up works can focus on how to improve the reasoning speed of the model， make the training and reasoning of the model glass， and use the method of emphasizing parameters for reference to further improve the model. Thus， LDHNet can be further competent for the needs of actual production and life.
Andriluka M ， Pishchulin L ， Gehler P and Schiele B . 2014 . 2D human pose estimation： new benchmark and state of the art analysis // Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus， USA ： IEEE： 3686 - 3693 ［ DOI： 10.1109/CVPR.2014.471 http://dx.doi.org/10.1109/CVPR.2014.471 ］
Cai Z D ， Ying N ， Guo C S ， Guo R and Yang P . 2021 . Research on multiperson pose estimation combined with YOLOv3 pruning model . Journal of Image and Graphics ， 26 （ 4 ）： 837 - 846
蔡哲栋 ， 应娜 ， 郭春生 ， 郭锐 ， 杨鹏 . 2021 . YOLOv3剪枝模型的多人姿态估计 . 中国图象图形学报 ， 26 （ 4 ）： 837 - 846 ［ DOI： 10.11834/jig.200138 http://dx.doi.org/10.11834/jig.200138 ］
Chen Y L ， Wang Z C ， Peng Y X ， Zhang Z Q ， Yu G and Sun J . 2018 . Cascaded pyramid network for multi-person pose estimation // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 7103 - 7112 ［ DOI： 10.1109/CVPR.2018.00742 http://dx.doi.org/10.1109/CVPR.2018.00742 ］
Cheng B W ， Xiao B ， Wang J D ， Shi H H ， Huang T S and Zhang L . 2020 . HigherHRNet： scale-aware representation learning for bottom-up human pose estimation // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 5385 - 5394 ［ DOI： 10.1109/CVPR42600.2020.00543 http://dx.doi.org/10.1109/CVPR42600.2020.00543 ］
Deng Y N ， Luo J X and Jin F L . 2019 . Overview of human pose estimation methods based on deep learning . Computer Engineering and Applications ， 55 （ 19 ）： 22 - 42
邓益侬 ， 罗健欣 ， 金凤林 . 2019 . 基于深度学习的人体姿态估计方法综述 . 计算机工程与应用 ， 55 （ 19 ）： 22 - 42 ［ DOI： 10.3778/j.issn.1002-8331.1906-0113 http://dx.doi.org/10.3778/j.issn.1002-8331.1906-0113 ］
Gao K ， Li W G ， Shu Y ， Wang Z G and Ge Y K . 2022 . Multi-scale lightweight human pose estimation with dense connections . Computer Engineering and Applications ， 58 （ 24 ）： 196 - 204
高坤 ， 李汪根 ， 束阳 ， 王志格 ， 葛英奎 . 2022 . 融入密集连接的多尺度轻量级人体姿态估计 . 计算机工程与应用 ， 58 （ 24 ）： 196 - 204 ［ DOI： 10.3778/j.issn.1002-8331.2208-0209 http://dx.doi.org/10.3778/j.issn.1002-8331.2208-0209 ］
Hou Q B ， Zhou D Q and Feng J S . 2021 . Coordinate attention for efficient mobile network design // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville， USA ： IEEE： 13708 - 13717 ［ DOI： 10.1109/CVPR46437.2021.01350 http://dx.doi.org/10.1109/CVPR46437.2021.01350 ］
Howard A G ， Zhu M L ， Chen B ， Kalenichenko D ， Wang W J ， Weyand T ， Andreetto M and Adam H . 2017 . MobileNets： efficient convolutional neural networks for mobile vision applications ［EB/OL］. ［ 2023-05-31 ］. https://arxiv.org/pdf/1704.04861.pdf https://arxiv.org/pdf/1704.04861.pdf
Huang G ， Liu Z ， Van Der Maaten L and Weinberger K Q . 2017 . Densely connected convolutional networks // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 2261 - 2269 ［ DOI： 10.1109/CVPR.2017.243 http://dx.doi.org/10.1109/CVPR.2017.243 ］
Iandola F N ， Han S ， Moskewicz M W ， Ashraf K ， Dally W J and Keutzer K . 2016 . SqueezeNet： AlexNet-level accuracy with 50 × fewer parameters and < 0 . 5 MB model size ［EB/OL］. ［ 2023-05-31 ］. https://arxiv.org/pdf/1602.07360.pdf https://arxiv.org/pdf/1602.07360.pdf
Kong Y H ， Qin Y F and Zhang K . 2023 . Deep learning based two-dimension human pose estimation： a critical analysis . Journal of Image and Graphics ， 28 （ 7 ）： 1965 - 1989
孔英会 ， 秦胤峰 ， 张珂 . 2023 . 深度学习二维人体姿态估计方法综述 . 中国图象图形学报 ， 28 （ 7 ）： 1965 - 1989 ［ DOI： 10.11834/jig.220436 http://dx.doi.org/10.11834/jig.220436 ］
Lee Y ， Hwang J W ， Lee S ， Baw Y and Park J . 2019 . An energy and GPU-computation efficient backbone network for real-time object detection // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Long Beach， USA ： IEEE： 752 - 760 ［ DOI： 10.1109/CVPRW.2019.00103 http://dx.doi.org/10.1109/CVPRW.2019.00103 ］
Li Q ， Zhang Z Y ， Xiao F ， Zhang F and Bhanu B . 2022 . Dite-HRNet： dynamic lightweight high-resolution network for human pose estimation ［EB/OL］. ［ 2023-05-31 ］. https://arxiv.org/pdf/2204.10762.pdf https://arxiv.org/pdf/2204.10762.pdf
Lin T Y ， Maire M ， Belongie S ， Hays J ， Perona P ， Ramanan D ， Dollr P and Zitnick C L . 2014 . Microsoft COCO： common objects in context // Proceedings of the 13th European Conference on Computer Vision . Zurich， Switzerland ： Springer： 740 - 755 ［ DOI： 10.1007/978-3-319-10602-1_48 http://dx.doi.org/10.1007/978-3-319-10602-1_48 ］
Luvizon D C ， Picard D and Tabia H . 2018 . 2D/3D pose estimation and action recognition using multitask deep learning // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 5137 - 5146 ［ DOI： 10.1109/CVPR.2018.00539 http://dx.doi.org/10.1109/CVPR.2018.00539 ］
Ma N N ， Zhang X Y ， Zheng H T and Sun J . 2018 . ShuffleNet V2： practical guidelines for efficient CNN architecture design // Proceedings of the 15th European Conference on Computer Vision （ECCV） . Munich， Germany ： Springer： 122 - 138 ［ DOI： 10.1007/978-3-030-01264-9_8 http://dx.doi.org/10.1007/978-3-030-01264-9_8 ］
Mehta S ， Rastegari M ， Shapiro L and Hajishirzi H . 2019 . ESPNetv2： a light-weight， power efficient， and general purpose convolutional neural network // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 9182 - 9192 ［ DOI： 10.1109/CVPR.2019.00941 http://dx.doi.org/10.1109/CVPR.2019.00941 ］
Newell A ， Yang K Y and Deng J . 2016 . Stacked hourglass networks for human pose estimation // Proceedings of the 14th European Conference on Computer Vision . Amsterdam， the Netherlands ： Springer： 483 - 499 ［ DOI： 10.1007/978-3-319-46484-8_29 http://dx.doi.org/10.1007/978-3-319-46484-8_29 ］
Sandler M ， Howard A ， Zhu M L ， Zhmoginov A and Chen L C . 2018 . MobileNetV2： inverted residuals and linear bottlenecks // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 4510 - 4520 ［ DOI： 10.1109/CVPR.2018.00474 http://dx.doi.org/10.1109/CVPR.2018.00474 ］
Sun K ， Xiao B ， Liu D and Wang J D . 2019 . Deep high-resolution representation learning for human pose estimation // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 5686 - 5696 ［ DOI： 10.1109/CVPR.2019.00584 http://dx.doi.org/10.1109/CVPR.2019.00584 ］
Wang J D ， Sun K ， Cheng T H ， Jiang B R ， Deng C R ， Zhao Y ， Liu D ， Mu Y D ， Tan M K ， Wang X G ， Liu W Y and Xiao B . 2021 . Deep high-resolution representation learning for visual recognition . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 43 （ 10 ）： 3349 - 3364 ［ DOI： 10.1109/TPAMI.2020.2983686 http://dx.doi.org/10.1109/TPAMI.2020.2983686 ］
Wang Q L ， Wu B G ， Zhu P F ， Li P H ， Zuo W M and Hu Q H . 2020 . ECA-Net： efficient channel attention for deep convolutional neural networks // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR） . Seattle， USA ： IEEE： 11531 - 11539 ［ DOI： 10.1109/CVPR42600.2020.01155 http://dx.doi.org/10.1109/CVPR42600.2020.01155 ］
Wang R J ， Li X and Ling C X . 2018 . Pelee： a real-time object detection system on mobile devices // Proceedings of the 32nd International Conference on Neural Information Processing Systems . Montréal， Canada ： Curran Associates Inc.： 1967 - 1976
Xiao B ， Wu H P and Wei Y C . 2018 . Simple baselines for human pose estimation and tracking // Proceedings of the 15th European Conference on Computer Vision （ECCV） . Munich， Germany ： Springer： 472 - 487 ［ DOI： 10.1007/978-3-030-01231-1_29 http://dx.doi.org/10.1007/978-3-030-01231-1_29 ］
Xu Y ， Zhang J ， Zhang Q M and Tao D C . 2022 . Vitpose： simple vision Transformer baselines for human pose estimation . Advances in Neural Information Processing Systems ， 2022， 35 ： 38571 - 38584
Yu C Q ， Xiao B ， Gao C X ， Yuan L ， Zhang L ， Sang N and Wang J D . 2021 . Lite-HRNet： a lightweight high-resolution network // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville， USA ： IEEE： 10435 - 10445 ［ DOI： 10.1109/CVPR46437.2021.01030 http://dx.doi.org/10.1109/CVPR46437.2021.01030 ］
Yuan Y H ， Fu R ， Huang L ， Lin W H ， Zhang C ， Chen X L and Wang J D . 2021 . HRFormer： high-resolution transformer for dense prediction ［EB/OL］. ［ 2023-05-31 ］. https://arxiv.org/pdf/2110.09408.pdf https://arxiv.org/pdf/2110.09408.pdf
Zhang Z ， Tang J and Wu G S . 2020 . Simple and lightweight human pose estimation ［EB/OL］. ［ 2023-05-31 ］. https://arxiv.org/pdf/1911.10346.pdf https://arxiv.org/pdf/1911.10346.pdf
相关文章
相关作者
相关机构
京公网安备11010802024621
