浏览全部资源
扫码关注微信
纸质出版日期： 2022-03-16 ，
录用日期： 2022-01-04
移动端阅览
张亚东, 汪玲, 兰海, 翟禹樵, 程洪. 面向小样本股骨骨折分型的多视角注意力融合方法[J]. 中国图象图形学报, 2022,27(3):784-796.
Yadong Zhang, Ling Wang, Hai Lan, Yuqiao Zhai, Hong Cheng. Multi-view attention fusion method for few-shot femoral fracture classification[J]. Journal of Image and Graphics, 2022,27(3):784-796.
目的
2
股骨粗隆间骨折是老年人最常见的骨折，不同类型的骨折需要不同的治疗方法。计算机图像识别技术可以辅助医生提高诊断准确率。传统的图像特征提取和机器学习方法，无法实现细粒度、高精度的分类，且少见针对3维图像的骨折分型方法。基于深度学习方法，通常需要大量的样本参与训练才能得出较好的分型性能。针对上述问题，本文提出一种面向小样本、多分类的骨折分型方法。
方法
2
将原始CT（computed tomography）分层扫描图像进行3维重建，获取不同视角下的2维图像信息，利用添加注意力机制的多视角深度学习网络融合组合特征，并联合旋转网络获得视角不变特征，最终得到预期分型结果。
结果
2
针对自建训练数据集（5类，每类23个样本），实验在4种3维深度学习网络模型上进行比较。基于注意力机制的多视角融合深度学习方法比传统深度学习模型的准确率提高了25%；基于旋转网络的方法比多视角深度学习方法提高8%。通过对比实验表明，提出的多视角融合深度学习方法大大优于传统基于体素的方法，并且也有利于使网络快速收敛。
结论
2
在骨折分型中，本文提出的添加注意力机制的多视角融合分型方法优于传统基于体素的深度学习方法，具有更高的准确率和更好的性能。
Objective
2
Femoral intertrochanteric fracture is the most common fracture in the elderly. Each type of fracture requires a specific treatment method. Computer imaging techniques
such as X-ray and computerized tomography (CT)
are used to help doctors in clinical diagnosis. Considering the complex fracture types and the large number of patients
missed diagnosis or misdiagnosis is incurred. In recent years
the development of computer image recognition technology has helped doctors improve the diagnostic accuracy. Femoral fractures have two types
namely
Arbeitsgemeinschaftfür Osteosynthesefragen(AO)/Orthopaedic Trauma Association(OTA) and six-types. The classification methods can be divided into traditional machine learning methods and deep learning methods. In traditional machine learning methods
man-made features are used for learning to make classification. However
these methods usually cannot achieve fine-grained and high-precision classification
and only a few fracture classification methods can be used for three-dimensional images. The deep learning method usually needs a large number of samples to participate in training to obtain good performance. To solve the above problems
this paper proposes a fracture classification method for small samples and multiple classification.
Method
2
An attention-based multi-view fusion network is proposed
in which a data-fusion strategy is used to improve the feature-fusion performance. Firstly
the original CT layered scanning images are reconstructed to three-dimension
and then two-dimensional images are obtained from different viewpoints. Secondly
a multi-view depth learning network with attention mechanism is used to fuse the different features with different viewpoints. Max-pooling
fully connective layer (FC) and rectified linear unit (ReLU) layers are used for learning the weights of different viewpoints. These layers are used to learn the view attention. The max-pooling operator down-sample the
H
×
W
×
M
original samples' tensor to 1×1×
M
which is then down-sampled to 1×1×
M/r
by the FC layer. The weighted parameters of each viewpoint are obtained using the ReLu and Sigmoid operations. Thirdly
the multiview images are multiplied by the view-weights and work as inputs of convolutional neural network (CNN). The probability that the sample falls into one class is learnt in the CNN. The attention mechanism helps network learning distinctive features. Moreover
the multi-view tensor reduces data dimension
thus improving CNN performance under small data sample size. With the consideration of CT scanning difference
pose changes are observed in 3D reconstructed models. These differences will result in uncertainty learning and reduce the classification performance. Then
a rotation network is used to obtain the view invariant features. RotationNet is defined as a differentiable multi-layer CNN
which has an additional viewpoint variable to learn how to compare with aforementioned multi-view network. The additional viewpoint variable functions to label incorrect view. The final layer of RotationNet is a concatenation of multi-view SoftMax layer
each of which outputs the category likelihood of each image. The category likelihood should be close to one when the estimated is correct. RotationNet only use partial set of multi-view images for classification
making it useful in typical scenarios
where only partial-view images are available. The RotationNet uses 2D CNN as backbone
in which large training sample size is needed. Then
in this paper
transfer learning is processed in the training step to improve the performance on multiple classification. The parameters of RotationNet are pre-trained on ModelNet40. A global parameter fine tuning process is employed on the fracture data in training step considering the difference of ModelNet40 and our fracture data.
Result
2
The proposed methods are compared with two three-dimensional deep learning network models
namely
3D ResNet and original multi-view CNN. Two types of classification
namely
AO and six-type
are used. A total of 23 training samples and 10 testing samples are present in each category. Firstly
the number of viewpoints is analyzed. Experimental results illustrate that the classification performance is improved when the number of viewpoints is changed from 4 to 12. However
the performance fluctuated when viewpoint number is great than 16. The reason is because of similarity between samples
which can be considered as same sample and results to performance reduce. In the following experiments
the number of viewpoints is set to 12. Secondly
the attention mechanism is analyzed. The proposed attention multi-view CNN (MV_att) is compared with original multi-view CNN (MVCNN) on the data-fusion model. The area under curve of our proposed MV_att is improved by approximately 3% on AO classification
which is approximately 5% in average on six-type classification. Thirdly
the performance of the models is analyzed. The accuracy of MV_att is 25% higher than that of MVCNN on AO classification. The pre-training RotationNet is 8% higher than MV_att on the six-type classification. Comparative experiments show that the proposed multiview fusion depth learning method is much better than the traditional voxel-based method
and it is also conducive to the rapid convergence of the network.
Conclusion
2
In fracture classification
the multi-view fusion classification method with attention mechanism proposed in this paper has higher accuracy than the traditional voxel depth learning method. The attention mechanism is useful in extracting distinct features. The multi-view data fusion model is useful in reducing the needs of sample size. The transfer learning is useful in improving the performance of the network.
骨折分型3维重建多视角采样多视角融合注意力机制
fracture classification3D reconstructionmulti-view samplingmulti-view fusionattention mechanism
Chen S H, Ma K and Zheng Y F. 2019. Med3D: transfer learning for 3D medical image analysis[EB/OL]. [2021-08-02].https://arxiv.org/pdf/1904.00625.pdfhttps://arxiv.org/pdf/1904.00625.pdf
Chen Z Y, Li K N and Zhang Z X. 2015. A finite element analysis of six-segment classification of femur intertrochanteric fracture. Chinese Journal of Orthopaedic Trauma, 17(5): 433-437
陈振沅, 李开南, 张之玺. 2015. 股骨转子间六部分骨折分型产生机制的有限元分析. 中华创伤骨科杂志, 17(5): 433-437[DOI:10.3760/cma.j.issn.1671-7600.2015.05.014]
Guo X W. 2014. The Study of Six Part Fracture Classification of Femoral Trochanteric Fractures and Trauma Scoring System (KNXW). Zunyi: Zunyi Medical College
郭小微. 2014. 股骨转子部骨折六部分骨折分型及创伤评分系统(KNXW)的研究. 遵义: 遵义医学院)[DOI: 10.7666/d.D535727http://dx.doi.org/10.7666/d.D535727]
Kanezaki A, Matsushita Y and Nishida Y. 2018. RotationNet: joint object categorization and pose estimation using multiviews from unsupervised viewpoints//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 5010-5019[DOI: 10.1109/CVPR.2018.00526http://dx.doi.org/10.1109/CVPR.2018.00526]
Klaber I, Besa P, Sandoval F, Lobos D, Zamora T, SchweitzerD and Urrutia J. 2021. The new AO classification system for intertrochanteric fractures allows better agreement than the original AO classification. An inter-and intra-observer agreement evaluation. Injury, 52(1): 102-105[DOI:10.1016/j.injury.2020.07.020]
Krizhevsky A, Sutskever I and Hinton G E. 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84-90[DOI:10.1145/3065386]
Lee C, Jang J, Lee S, Kim Y S, Jo H J and Kim Y. 2020. Classification of femur fracture in pelvic X-ray images using meta-learned deep neural network. Scientific Reports, 10(1): #13694[DOI:10.1038/s41598-020-70660-4]
Li J T, Tang S J, Zhang H, Li Z R, Deng W Y, Zhao C, Fan L H, Wang G Q, Liu J H, Yin P, Xu G X, Zhang L C and Tang P F. 2019. Clustering of morphological fracture lines for identifying intertrochanteric fracture classification with hausdorff distance-based K-means approach. Injury, 50(4): 939-949[DOI:10.1016/j.injury.2019.03.032]
Maicas G, Bradley A P, Nascimento J C, Reid I and Carneiro G. 2018. Training medical image analysis systems like radiologists//Proceedings of the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention. Granada, Spain: Springer: 546-554[DOI: 10.1007/978-3-030-00928-1_62http://dx.doi.org/10.1007/978-3-030-00928-1_62]
Marsh J L, Slongo T F, Agel J, Broderick J S, Creevey W, Decoster T A, Prokuski L, Sirkin M S, Ziran B, Henley B and Audigé L D V M. 2007. Fracture and dislocation classification compendium-2007: orthopaedic trauma association classification, database and outcomes committee. Journal of Orthopaedic Trauma, 21(S10): S1-S6[DOI:10.1097/00005131-200711101-00001]
Mutasa S, Varada S, Goel A, Wong T T and Rasiej M J. 2020. Advanced deep learning techniques applied to automated femoral neck fracture detection and classification. Journal of Digital Imaging, 33(5): 1209-1217[DOI:10.1007/s10278-020-00364-8]
Olczak J, Emilson F, Razavian A, Antonsson T, Stark A and Gordon M. 2021. Ankle fracture classification using deep learning: automating detailed AO Foundation/Orthopedic Trauma Association (AO/OTA) 2018 malleolar fracture identification reaches a high degree of correct classification. Acta Orthopaedica, 92(1): 102-108[DOI:10.1080/17453674.2020.1837420]
Sangeetha S, Sujatha C M and Manamalli D. 2014. Anisotropic analysis of trabecular architecture in human femur bone radiographs using quaternion wavelet transforms//Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Chicago, USA: IEEE: 5603-5606[DOI: 10.1109/EMBC.2014.6944897http://dx.doi.org/10.1109/EMBC.2014.6944897]
Su H, Maji S, Kalogerakis E and Learned-Miller E. 2015. Multi-view convolutional neural networks for 3D shape recognition//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 945-953[DOI: 10.1109/ICCV.2015.114http://dx.doi.org/10.1109/ICCV.2015.114]
Tang C S, Hu C C, Sun J D and Sima H F. 2021. Deep learning-based medical images analysis evolved from convolution to graph convolution. Journal of Image and Graphics, 26(9): 2078-2093
唐朝生, 胡超超, 孙君顶, 司马海峰. 2021. 医学图像深度学习技术: 从卷积到图卷积的发展. 中国图象图形学报, 26(9): 2078-2093[DOI:10. 11834/jig. 200666]
Tripathi A M, Upadhyay A, Rajput A S, Singh A P and Kumar B. 2017. Automatic detection of fracture in femur bones using image processing//Proceedings of 2017 International Conference on Innovations in Information, Embedded and Communication Systems. Coimbatore, India: IEEE: 1-5[DOI: 10.1109/ICIIECS.2017.8275843http://dx.doi.org/10.1109/ICIIECS.2017.8275843]
Wada K, Mikami H, Toki S, Amari R, Takai M and Sairyo K. 2020. Intra-and inter-rater reliability of a three-dimensional classification system for intertrochanteric fracture using computed tomography. Injury, 51(11): 2682-2685[DOI:10.1016/j.injury.2020.07.047]
Wang L, Cheng H, Lan H, Zheng Y J and Li K N. 2016. Automatic recognition of pertrochanteric bone fractures in femur using level sets//Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Orlando, USA: IEEE: 3851-3854[DOI: 10.1109/EMBC.2016.7591568http://dx.doi.org/10.1109/EMBC.2016.7591568]
Xu K and Li K N. 2019. A finite element analysis of fixation with proximal femoral nail antirotation, dynamic hip screw and percutaneous compression plate for six-part intertrochanteric fractures. Chinese Journal of Orthopaedic Trauma, 21(4): 345-352
徐锴, 李开南. 2019. 三种内固定固定股骨转子间六部分骨折各分型稳定性的有限元分析. 中华创伤骨科杂志, 21(4): 345-352[DOI:10.3760/cma.j.issn.1671-7600.2019.04.013]
Yin B, He Y M, Wang D and Zhou J L. 2021. Classification of femur trochanteric fracture: evaluating the reliability of Tang classification. Injury, 52(6): 1500-1505[DOI:10.1016/j.injury.2020.11.031]
Yoon S J, Kim T H, Joo S B and Oh S E. 2020. Automatic multi-class intertrochanteric femur fracture detection from CT images based on AO/OTA classification using faster R-CNN-BO method. Journal of Applied Biomedicine, 18(4): 97-105[DOI:10.32725/jab.2020.013]
Zhao X, Peng H and Li S B. 2020. Application of spiral CT three-dimensional reconstruction and X-ray in diagnosis and treatment of hip fracture. Shanghai of Biomedical Engineering, 41(2): 95-97
赵旭, 彭弘, 李圣博. 2020. 螺旋CT3维重建与X线在髋关节骨折诊断及治疗中的应用. 生物医学工程学进展, 41(2): 95-97[DOI:10.3969/j.issn.1674-1242.2020.02.009]
Zuo Y, Huang G and Nie S D. 2021. Application and challenges of deep learning in the intelligent processing of medical images. Journal of Image and Graphics, 26(2): 0305-0315
左艳, 黄钢, 聂生东. 2021. 深度学习在医学影像智能处理中的应用与挑战. 中国图象图形学报, 26(2): 0305-0315[DOI:10. 11834/jig. 190470]
相关作者
相关机构
京公网安备11010802024621