浏览全部资源
扫码关注微信
收稿：2022-06-20，
修回：2022-09-22，
纸质出版：2023-09-16
移动端阅览
目的
2
基于单幅RGB图像的手势姿态估计受手势复杂性、手指特征局部自相似性及遮挡问题的影响，导致手势姿态估计准确率低。为此，提出一种面向单目视觉手势姿态估计的多尺度特征融合网络。
方法
2
1）采用ResNet50（50-layer residual network）模块从RGB图像提取不同分辨率特征图，通过通道变换模块显式地学习特征通道间的依赖关系，增强重要的特征通道信息，弱化次要的特征通道信息。2）在全局回归模块中，通过设计节点间的连接方式融合不同分辨率特征图，以便充分利用图像的细节与整体信息。采用局部优化模块继续提取更深层的特征信息，获得手部关节点的高斯热图，以此修正遮挡等原因造成部分关节点回归不准确的问题。3）计算经通道变换模块处理后的最小特征图，通过全局池化和多层感知机处理该特征图以获得手势类别和右手相对于左手的深度。4）综合以上结果获得最终的手势姿态。
结果
2
采用InterHand2.6M和RHD（rendered handpose dataset）数据集训练多尺度特征融合网络，评估指标中根节点的平均误差和关节点的平均误差，均低于同类方法，且在一些复杂和遮挡的场景下鲁棒性更高。在InterHand2.6M数据集上，与InterNet方法相比，本文方法的交互手关节点的平均误差降低5.8%，单手关节点的平均误差降低8.3%，根节点的平均误差降低5.1%。从RHD数据集的测试结果看，与同类方法相比，本文方法在手部关节点的平均误差上获得最小值。
结论
2
本文提出的多尺度特征融合网络能够更准确地预测手部关节点位置，适用于复杂手势或遮挡条件下的手势姿态估计（本文方法代码网址：
https：//github.com/cornersInHeart/hand-pose-esitmation.git
https://github.com/cornersInHeart/hand-pose-esitmation.git
）。
Objective
2
Hand pose estimation aims to identify and localize key points of human hands in images. It has a wide range of applications in computer vision. Hand pose estimation methods can be categorized as depth- or RGB-based methods. Depth-based methods estimate the hand pose by extracting depth features. They require specific devices to constrain the user environment. Scholars use RGB images for hand pose estimation. However， this approach is difficult in an occluded environment. In particular， hand pose estimation based on a single RGB image has low accuracy because of the complexity of the pose， local self-similarity of finger features， and occlusion. Edge information is usually ignored in hand pose estimation. However， this information is important in extracting the information of occluded parts. Moreover， fingertips are small， thereby complicating the recognition of the joints at the fingertips. However， many existing RGB-based gesture estimation methods do not make good use of edge information. A multiscale feature fusion network for monocular vision gesture pose estimation is proposed to address this problem.
Method
2
Gesture pictures usually contain complex detailed features. A strong correlation between fingers and joints is present. Therefore， the use of a single feature for hand pose estimation tends to ignore diverse feature information， thereby complicating the accurate extraction of gesture information. Multi-scale feature fusion network （MS-FF） aims to estimate the hand pose through a single RGB image. The feature maps of different resolutions are extracted from RGB images through the ResNet50 module. Feature maps are fed into the channel conversion module to learn the dependencies between channels explicitly， thereby enhancing important information and downplaying minor information. The level of feature information depends on the resolution of a feature map. Thus， the global regression module obtains high-resolution feature maps containing semantic information. These maps are separately input in the local optimization module to extract deep information. The Gaussian heatmap of hand joints is obtained to improve the spatial generalization ability of the model. Thus， accurate joint locations can be obtained. We take the feature map with the smallest resolution from the channel conversion module， through which the handedness and relative depth information between the wrist joints are obtained. The above results are combined to estimate the hand pose.
Result
2
The PyTorch framework was used for training. The hand image was resized to 256 × 256 pixels and input to the network. In the experiment， the batch size was set to 16. The network was trained for 20 epochs with an NVIDIA 3090 GPU. The initial learning rate was set to 0.000 1 and reduced by a factor of 10 at the 15th and 17th epochs to optimize the network output. The proposed method achieved better metrics than other methods on different test sets. InterHand2.6M （H+M） was selected as the training set. Compared with the evaluation metrics obtained by InterNet， the mean relative root position error， mean per joint position error of single hand sequences， and mean per joint position error of interacting hand sequences obtained by MS-FF had low errors of 30.92， 11.10， and 15.14， respectively. These values were 5.1%， 8.3%， and 5.8% lower than those obtained by InterNet. We also found that each finger achieved a low error. MS-FF also possesses few model parameters and low computational complexity while improving recognition accuracy. However， the running rate of MS-FF （28 frame/s） is lower than that of InterNet （53 frame/s）. The picture shows the hand pose with finger self-occlusion and mutual occlusion of hands. Thus， estimating this interacting hand pose is more difficult than predicting a single hand pose. In the result obtained by our method， the hand joint positions and hand pose estimations are correctly predicted under occlusion. Moreover， our method can accurately predict hand joint positions and hand poses in case of occlusion. The proposed method achieves good recognition results in occluded gestures.
Conclusion
2
This study proposes an MS-FF for monocular visual hand pose estimation. MS-FF can extract information of different levels from feature maps of different resolutions to process the detailed information of occluded edges and fingertips effectively and estimate hand poses accurately. MS-FF accurately estimates hand poses in an RGB image and copes well with complex application scenarios. Thus， it can deal with difficult-to-recognize joints and inaccurate gesture recognition in occlusion scenes. Channels contain various implicit information. We need to focus on the information that is important for recognizing gestures. A channel conversion module adjusts the weights of channels to enhance important information. Fingertips occupy a small percentage of an image. They are also relatively difficult to identify. A global regression module generates different resolutions with rich semantic information to utilize image edge details and deep information effectively. This module is important in estimating finger poses. The global regression module may not accurately identify occluded joints. A local optimization module is designed with deep information in the feature map. It fuses all-level feature maps by correcting joints that do not return to the correction position. Thus， these maps can be applied well to the occlusion scene. Our method can effectively estimate single and interacting hand poses. It can also avoid errors caused by occlusion to a certain extent. High accuracy and robustness are achieved using the proposed method. However， the running rate of MS-FF is slower than that of the InterNet method because of the complex construction process of the MS-FF method. This scenario increases serial wait， kernel startup， and synchronization time overhead. In future work， we will continue to optimize our model， reduce the running rate of the model while ensuring recognition accuracy， and achieve a fast recognition speed to pave the way for fast and accurate gesture recognition in real scenes.
Baek S ， Kim K I and Kim T K . 2019 . Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 1067 - 1076 ［ DOI： 10.1109/CVPR.2019.00116 http://dx.doi.org/10.1109/CVPR.2019.00116 ］
Bao W X ， Xie D W ， Zhu M and Liang D . 2019 . Gesture recognition based on aggregate channel feature and dual-tree complex wavelet transform . Journal of Image and Graphics ， 24 （ 7 ）： 1067 - 1075
鲍文霞 ， 解栋文 ， 朱明 ， 梁栋 . 2019 . 结合聚合通道特征和双树复小波变换的手势识别 . 中国图象图形学报 ， 24 （ 7 ）： 1067 - 1075 ［ DOI： 10.11834/jig.180504 http://dx.doi.org/10.11834/jig.180504 ］
Cai Y J ， Ge L H ， Ca J F and Yuan J S . 2018 . Weakly-supervised 3D hand pose estimation from monocular RGB images // Proceedings of the 15th European Conference on Computer Vision . Munich， Germany ： Springer： 678 - 694 ［ DOI： 10.1007/978-3-030-01231-1_41 http://dx.doi.org/10.1007/978-3-030-01231-1_41 ］
Chen L J ， Lin S Y ， Xie Y S ， Lin Y Y ， Fan W and Xie X H . 2020a . DGGAN： depth-image guided generative adversarial networks for disentangling RGB and depth images in 3D hand pose estimation // Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass， USA ： IEEE： 400 - 408 ［ DOI： 10.1109/WACV45572.2020.9093380 http://dx.doi.org/10.1109/WACV45572.2020.9093380 ］
Chen L J ， Lin S Y ， Xie Y S ， Tang H ， Xue Y F ， Xie X H ， Lin Y Y and Fan W . 2019 . Generating realistic training images based on tonality-alignment generative adversarial networks for hand pose estimation // Proceedings of the 30th British Machine Vision Conference . Cardiff， UK ： #09916 ［ DOI： 10.48550/arXiv.1811.09916 http://dx.doi.org/10.48550/arXiv.1811.09916 ］
Chen X H ， Wang G J ， Guo H K and Zhang C R . 2020b . Pose guided structured region ensemble network for cascaded hand pose estimation . Neurocomputing ， 395 ： 138 - 149 ［ DOI： 10.1016/j.neucom.2018.06.097 http://dx.doi.org/10.1016/j.neucom.2018.06.097 ］
Chen Y J ， Tu Z G ， Kang D ， Bao L C ， Zhang Y ， Zhe X F ， Chen R Z and Yuan J . 2021 . Model-based 3D hand reconstruction via self-supervised learning // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR） . Nashville， USA ： IEEE： 10446 - 10455 ［ DOI： 10.1109/CVPR46437.2021.01031 http://dx.doi.org/10.1109/CVPR46437.2021.01031 ］
Fan L L ， Rao H and Yang W J . 2021 . 3D hand pose estimation based on five-layer ensemble CNN . Sensors ， 21 （ 2 ）： # 649 ［ DOI： 10.3390/s21020649 http://dx.doi.org/10.3390/s21020649 ］
Ge L H ， Ren Z ， Li Y C ， Xue Z H ， Wang Y Y ， Cai J F and Yuan J S . 2019 . 3D hand shape and pose estimation from a single RGB image // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 10825 - 10834 ［ DOI： 10.1109/CVPR.2019.01109 http://dx.doi.org/10.1109/CVPR.2019.01109 ］
Iqbal U ， Molchanov P ， Breuel T ， Gall J and Kautz J . 2018 . Hand pose estimation via latent 2.5D heatmap regression // Proceedings of the 15th European Conference on Computer Vision . Munich， Germany ： Springer： 125 - 143 ［ DOI： 10.1007/978-3-030-01252-6_8 http://dx.doi.org/10.1007/978-3-030-01252-6_8 ］
Ishii A ， Nakano G and Inoshita T . 2021 . Occlusion-robust 3D hand pose estimation from a single RGB image // Proceedings of the 17th International Conference on Machine Vision and Applications （MVA） . Aichi， Japan ： IEEE： #9511389 ［ DOI： 10.23919/MVA51890.2021.9511389 http://dx.doi.org/10.23919/MVA51890.2021.9511389 ］
Kong D Y ， Ma H Y ， Chen Y F and Xie X H . 2020 . Rotation-invariant mixed graphical model network for 2D hand pose estimation // Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass， USA ： IEEE： 1535 - 1544 ［ DOI： 10.1109/WACV45572.2020.9093638 http://dx.doi.org/10.1109/WACV45572.2020.9093638 ］
Liu S W ， Wang G J ， Xie P W and Zhang C R . 2020 . Light and fast hand pose estimation from spatial-decomposed latent heatmap . IEEE Access ， 8 ： 53072 - 53081 ［ DOI： 10.1109/ACCESS.2020.2979507 http://dx.doi.org/10.1109/ACCESS.2020.2979507 ］
Ma N N ， Zhang X Y ， Zheng H T and Sun J . 2018 . ShuffleNet V2： practical guidelines for efficient CNN architecture design // Proceedings of the 15th European Conference on Computer Vision . Munich， Germany ： Springer： 122 - 138 ［ DOI： 10.1007/978-3-030-01264-9_8 http://dx.doi.org/10.1007/978-3-030-01264-9_8 ］
Moon G ， Yu S I ， Wen H ， Shiratori T and Lee K M . 2020 . InterHand2.6M： A dataset and baseline for 3D interacting hand pose estimation from a single RGB image // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 548 - 564 ［ DOI： 10.1007/978-3-030-58565-5_33 http://dx.doi.org/10.1007/978-3-030-58565-5_33 ］
Mueller F ， Bernard F ， Sotnychenko O ， Mehta D ， Sridhar S ， Casas D and Theobalt C . 2018 . GANerated hands for real-time 3D hand tracking from monocular RGB // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 49 - 59 ［ DOI： 10.1109/CVPR.2018.00013 http://dx.doi.org/10.1109/CVPR.2018.00013 ］
Panteleris P ， Oikonomidis I and Argyros A . 2018 . Using a single RGB frame for real time 3D hand pose estimation in the wild // Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision . Lake Tahoe， USA ： IEEE： 436 - 445 ［ DOI： 10.1109/WACV.2018.00054 http://dx.doi.org/10.1109/WACV.2018.00054 ］
Simon T ， Joo H ， Matthews I and Sheikh Y . 2017 . Hand keypoint detection in single images using multiview bootstrapping // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 4645 - 4653 ［ DOI： 10.1109/CVPR.2017.494 http://dx.doi.org/10.1109/CVPR.2017.494 ］
Spurr A ， Song J ， Park S and Hilliges O . 2018 . Cross-modal deep variational hand pose estimation // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 89 - 98 ［ DOI： 10.1109/CVPR.2018.00017 http://dx.doi.org/10.1109/CVPR.2018.00017 ］
Tompson J ， Stein M ， Lecun Y and Perlin K . 2014 . Real-time continuous pose recovery of human hands using convolutional networks . ACM Transactions on Graphics ， 33 （ 5 ）： # 169 ［ DOI： 10.1145/2629500 http://dx.doi.org/10.1145/2629500 ］
Wang Y ， Chen Y L and Sun Q L . 2021 . Hand gesture recognition in complex background . Journal of Image and Graphics ， 26 （ 4 ）： 815 - 827
王银 ， 陈云龙 ， 孙前来 . 2021 . 复杂背景下的手势识别 . 中国图象图形学报 ， 26 （ 4 ）： 815 - 827 ［ DOI： 10.11834/jig.200211 http://dx.doi.org/10.11834/jig.200211 ］
Yang L L and Yao A . 2019 . Disentangling latent hands for image synthesis and pose estimation // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 9869 - 9878 ［ DOI： 10.1109/CVPR.2019.01011 http://dx.doi.org/10.1109/CVPR.2019.01011 ］
Zhang M H ， Li C ， Wang Y C and Chen K Y . 2021a . HGposeGUCN： a lightweight network for 3D hand pose estimation from a single RGB image // Proceedings of the 6th IEEE International Conference on Signal and Image Processing . Nanjing， China ： IEEE： 478 - 482 ［ DOI： 10.1109/ICSIP52628.2021.9688661 http://dx.doi.org/10.1109/ICSIP52628.2021.9688661 ］
Zhang X Q ， Huang S L and Ye Z F . 2021b . Accurate 3D hand pose estimation network utilizing joints information . Signal Processing： Image Communication ， 90 ： # 116035 ［ DOI： 10.1016/j.image.2020.116035 http://dx.doi.org/10.1016/j.image.2020.116035 ］
Zimmermann C and Brox T . 2017 . Learning to estimate 3D hand pose from single RGB images // Proceedings of 2017 IEEE International Conference on Computer Vision . Venice， Italy ： IEEE： 4913 - 4921 ［ DOI： 10.1109/ICCV.2017.525 http://dx.doi.org/10.1109/ICCV.2017.525 ］
相关作者
相关机构
京公网安备11010802024621
