多分支深度特征融合的中医脑卒中辅助诊断

王麒达; 冀伦文; 强彦; 王华虎; 赵琛琦; 李慧芝; 赵紫娟

doi:10.11834/jig.210744

中医图像 | 浏览量 : 0 下载量: 270 CSCD: 0

PDF
导出
分享
收藏
专辑

多分支深度特征融合的中医脑卒中辅助诊断
Multi-branch deep feature fusion method for traditional Chinese medicine intervened human cerebral stroke aided diagnosis
2022年27卷第3期页码：935-947
收稿：2021-08-20，

修回：2021-11-23，

录用：2021-11-30，

纸质出版：2022-03-16
DOI： 10.11834/jig.210744
稿件说明：

移动端阅览

王麒达, 冀伦文, 强彦, 王华虎, 赵琛琦, 李慧芝, 赵紫娟. 多分支深度特征融合的中医脑卒中辅助诊断[J]. 中国图象图形学报, 2022,27(3):935-947. DOI： 10.11834/jig.210744.

Qida Wang, Lunwen Ji, Yan Qiang, Huahu Wang, Chenqi Zhao, Huizhi Li, Zijuan Zhao. Multi-branch deep feature fusion method for traditional Chinese medicine intervened human cerebral stroke aided diagnosis[J]. Journal of Image and Graphics, 2022, 27(3): 935-947. DOI： 10.11834/jig.210744.

摘要

目的

脑卒中发病征兆不明显，发病速度快且致死率高，目前医学领域的主要诊疗手段仍是针对脑卒中病时和病后，但在病前预测方面缺少有效办法。中医对于治未病等效果显著，其中望诊更是中医诊疗的重要方式。本文结合中医望诊，基于面部与手部图像提出了一种多分支深度特征融合的中医脑卒中辅助诊断方法。

方法

针对不同部位图像，分别构建两部位的双分支特征提取模块，将面部和手部的重点区域作为主体分支提取主要特征。根据中医望诊在面部与手部诊疗的特点，进一步将眉心的纹理特征和掌心的颜色特征作为辅助信息提取辅助特征；在此基础上提出信息交互模块(information interaction module，IIM)，将主要特征与辅助特征进行有效信息交互，从而辅助主体分支提取更有区别性的信息；最终将两部位的特征进行融合降维用于脑卒中辅助诊断。

结果

本文将采集的3 011例面部和手部图像数据进行筛查扩充后作为实验数据集，并在不同评价指标下与当前主流的分类模型进行对比。实验结果表明，本文方法在准确性上达到了83.36%，相比ResNet-34、DenseNet121、VGG16(Visual Geometry Group 16-layer net)和InceptionV3等其他主流分类模型性能提高了3%~7%；在特异性和敏感性方面分别为82.47% 和85.10%，其效果优于对比方法。

结论

本文方法能够有效结合中医望诊的诊疗经验并通过传统面部和手部图像实现对脑卒中的有效预测，为中医望诊在脑卒中方面的客观化和便捷化发展提供了帮助。

Abstract

Objective

Stroke is a severe human cerebrovascular disease that causes brain tissue damage due to sudden rupture of blood vessels or vascular obstruction originated blood flow inefficiency. The incidence of ischemic stroke is high frequency

high recurrence rate and high fatality rate. Traditional Chinese medicine (TCM) has its priority of stroke. In the four-diagnosis-inspection

listening

asking and feeling the pulse-(in Chinese) of Chinese medicine

the prime step of inspection shows that information extraction is one of the essential factors of TCM. But

TCM has its constraints of medical standardization and manual factor issues. A deep learning technology is benefit to further recognize the constraints of TCM.

Method

Our research illustrates a dual-branch cross-attention feature fusion model (DCFFM) based on facial images and hand images. It can assists in predicting stroke disease well. The overall model is divided into three parts: facial feature extraction module

hand feature extraction module and feature fusion module. For the facial feature extraction module

we construct the subject branch and auxiliary information branch to extract facial features. In accordance with the guidance of Chinese medicine doctors

we pre-process the facial image and use the key diagnostic area of the stroke in the face as the input of the main branch. In addition

we also integrate the knowledge of inspection of TCM to cut out the image of the area around the eyebrows

and use the Sobel filter to extract the gradient image as the input of the auxiliary information branch. For the hand feature extraction module

this demonstration adopts the same double-branch structure to cut out the palm area as the input of the main branch. In order to more stably and accurately reflect the pathological condition of the hand and reflect the small changes in the characteristics of the hand

we convert the palm image from the RGB color space to the HSV color space and transfer it to the differentiated auxiliary information branch. The proposed branches are respectively input to their respective convolution blocks

and the depth characteristics of the input data are extracted based on the convolution operation. Max pooling is used on the feature maps and batch normalization is used to prevent the model from over-fitting. In addition

we use two loss functions to constrain the training of the two feature extraction modules

and use the total loss to constrain the entire model. Between the two branches of each feature extraction module

we built an information interaction module (IIM) for further information interaction amongst the branches to reveal the model extract distinctive features. It assigns a certain weight to the feature map of auxiliary information and then interconnected with the feature of the subject branch. We use 1 × 1 convolution fusion to reduce dimensionality. Under no special operations circumstances

the IIM can be trained in an end-to-end manipulation. For the feature fusion module

multiple convolutional layers are used for overall fusion dimensionality reduction to generate the prediction result via the multi-branch deep feature fusion based on fusing the depth features of the facial feature extraction module and the hand feature extraction module.

Result

In order to aid model training and improve the stability and robustness of the model

our demonstration screens and extends the collected 3 011 face and hand image data. We remove some scrambled images with scars and conduct the data extension by horizontal flipping. We remove some images of peeling

disability

background clutter and implement random option for the remaining images to expand the data by horizontal flipping. It is determined that 3 964 images of positive and negative samples are involved as the experimental data set. Our multiple sets of comparative experiments and ablation experiments have been facilitated based on a variety of evaluation indicators to verify the performance of the model

such as accuracy

specificity

sensitivity

and F1-score. First

we compare the overall performance of the proposed method with the current mainstream classification algorithms. Experimental results show that the accuracy of the method proposed reaches 83.36%

which is 3%—7% higher than the performance of other mainstream classification models. Based on ten-fold cross-validation

the specificity and sensitivity reached 82.47% and 85.10% respectively. The illustrated sensitivity shows a relatively large advantage

indicating that the method in this paper has a better performance for the detection of true positives. Next

we still verified the impact of facial feature extraction module

hand feature extraction module and IIM on the performance of the model. This analyzed results show that feature extraction of face data and hand data can effectively improve the performance of the model simultaneously. In addition

the IIM has targeted the sensitivity and specificity of the model.

Conclusion

Our method can use human facial features and human hand data to assist in stroke prediction

and has good stability and robustness. Meanwhile

the demonstrated IIM also promotes the information interaction between multi-branch tasks.

关键词

Keywords

references

Cheng H D, Jiang X H, Sun Y and Wang J L. 2001. Color image segmentation: advances and prospects. Pattern Recognition, 34(12): 2259-2281 [DOI: 10.1016/S0031-3203(00)00149-7]

Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau N G, Venugopal V K,Mahajan V, Rao P and Warier P. 2018. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet, 392(10162): 2388-2396 [DOI: 10.1016/S0140-6736(18)31645-3]

Cho J, Park K S, Karki M, Lee E, Ko S, Kim J K, Lee D, Choe J, Son J, Kim M, Lee S, Lee J, Yoon C and Park S. 2019. Improving sensitivity on identification and delineation of intracranial hemorrhage lesion using cascaded deep learning models. Journal of Digital Imaging, 32(3): 450-461 [DOI: 10.1007/s10278-018-00172-1]

Cipolla R, Gal Y and Kendall A. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7482-7491 [ DOI: 10.1109/CVPR.2018.00781 http://dx.doi.org/10.1109/CVPR.2018.00781 ]

Feigin V L, Krishnamurthi R V, Parmar P, Norrving B, Mensah G A, Bennett D A, Barker-Collo S, Moran A E, Sacco R L, Truelsen T, Davis S, Pandian J D, Naghavi M, Forouzanfar M H, Nguyen G, Johnson C O, Vos T, Meretoja A, Murray C J L, Roth G A, GBD 2013 Writing Group and GBD 2013 Stroke Panel Experts Group. 2015. Update on the global burden of ischemic and hemorrhagic stroke in 1990-2013: the GBD 2013 study. Neuroepidemiology, 45(3): 161-176 [DOI: 10.1159/000441085]

Han X J, Gao Y R, Liu D S, Yu X and Ren C. 2018. Research on the theory and mechanism of three-dimensional observation diagnosis of coronary heart disease. Chinese Journal of Basic Medicine in Traditional Chinese Medicine, 24(4): 441-442, 456

韩学杰, 郜亚茹, 刘大胜, 于潇, 任聪. 2018. 冠心病大鱼际三维望诊法的理论与机制探究. 中国中医基础医学杂志, 24(4): 441-442, 456

He K M, Zhang X Y, Ren S Q and Jian S. 2016. Identity mappings in deep residual networks//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 630-645 [ DOI: 10.1007/978-3-319-46493-0_38 http://dx.doi.org/10.1007/978-3-319-46493-0_38 ]

Ioffe S and Szegedy C. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift//Proceedings of the 32nd International Conference on International Conference on Machine Learning. Lille, France: ICML: 448-456

Kim B H, Lee S H, Cho D U and Oh S Y. 2008. A proposal of heart diseases diagnosis method using analysis of face color//Proceedings of 2008 International Conference on Advanced Language Processing and Web Information Technology. Dalian, China: IEEE: 220-225 [ DOI: 10.1109/ALPIT.2008.27 http://dx.doi.org/10.1109/ALPIT.2008.27 ]

Kuo W C, Häne C, Mukherjee P, Malik J and Yuh E L. 2019. Expert-level detection of acute intracranial hemorrhage on head computed tomography using deep learning. Proceedings of the National Academy of Sciences of the United States of America, 116(45): 2737-22745 [DOI: 10.1073/pnas.1908021116]

Li B C, Huang Q M, Lu Y, Chen S H, Liang R and Wang Z P. 2007. A method of classifying tongue colors for traditional Chinese medicine diagnosis based on the CIELAB color space//Proceedings of the 1st International Conference on Medical Biometrics. Hong Kong, China: Springer: 153-159 [ DOI: 10.1007/978-3-540-77413-6_20 http://dx.doi.org/10.1007/978-3-540-77413-6_20 ]

Li J X, Zhang B, Lu G M, You J and Zhang D. 2019. Body surface feature-based multi-modal learning for diabetes mellitus detection. Information Sciences. 472: 1-14[DOI: 10.1016/j.ins.2018.09.010].

Lin M, Chen Q and Yan S C. 2013. Network in network [EB/OL]. [2021-08-10] . http://arxiv.org/pdf/1312.4400v3.pdf http://arxiv.org/pdf/1312.4400v3.pdf

Lin S, Li Z G, Fu B W, Chen S P, Li X, Wang Y, Wang X Y, Lv B, Xu B, Song X T, Zhang Y J, Cheng X, Huang W J, Pu J, Zhang Q, Xia Y L, Du B, Ji X Y and Zheng Z. 2020. Feasibility of using deep learning to detect coronary artery disease based on facial photo. European Heart Journal, 41(46): 4400-4411 [DOI: 10.1093/eurheartj/ehaa640]

Lu X G, Ma K W, Wang Y Q and Hao Y M. 2019. Study on tongue color features of healthy people with "Joy" emotion based on objectification of tongue diagnosis. World Science and Technology-Modernization of Traditional Chinese Medicine, 21(3): 471-476

鲁晓篝, 马凯雯, 王忆勤, 郝一鸣. 2019. 基于舌诊客观化的情志"喜"舌色特征研究. 世界科学技术-中医药现代化, 21(3): 471-476 [DOI: 10.11842/wst.2019.03.022]

Sobel I and Feldman G. 1973. A 3×3 isotropic gradient operator for image processing. Pattern Classification and Scene Analysis, 271-272

Su W, Xu Z Y, Wang Z Q and Xu J T. 2011. Objectified study on tongue images of patients with lung cancer of different syndromes. Chinese Journal of Integrative Medicine, 17(4): 272-276 [DOI: 10.1007/s11655-011-0702-6]

Wang D M, Zhang D and Lu G M. 2017a. Generalized feature extraction for wrist pulse analysis: from 1-D time series to 2-D matrix. IEEE Journal of Biomedical and Health Informatics, 21(4): 978-985 [DOI: 10.1109/JBHI.2016.2628238]

Wang F, Jiang M Q, Qian C, Yang S, Li C, Zhang H G, Wang X G and Tang X O. 2017b. Residual attention network for image classification//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 6450-6458 [ DOI: 10.1109/CVPR.2017.683 http://dx.doi.org/10.1109/CVPR.2017.683 ]

Wang P, Zuo W M and Zhang D. 2014. A compound pressure signal acquisition system for multichannel wrist pulse signal analysis. IEEE Transactions on Instrumentation and Measurement, 63(6): 1556-1565 [DOI: 10.1109/TIM.2013.2267458]

Yan K, Zhang D, Wu D R, Wei H and Lu G M. 2014. Design of a breath analysis system for diabetes screening and blood glucose level prediction. IEEE Transactions on Biomedical Engineering, 61(11): 2787-2795 [DOI: 10.1109/TBME.2014.2329753]

Zhang D, Zhang H and Zhang B. 2017. Tongue Shape Classification by Geometric Features. Tongue Image Analysis. Springer, Singapore. 133-153. [ DOI: 10.1007/978-981-10-2167-1_8 http://dx.doi.org/10.1007/978-981-10-2167-1_8 ]

Zhang L, Samaras D, Tomasi D, Volkow N and Goldstein R. 2005. Machine learning for clinical diagnosis from functional magnetic resonance imaging//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, USA: IEEE: 1211-1217 [ DOI: 10.1109/CVPR.2005.219 http://dx.doi.org/10.1109/CVPR.2005.219 ]

Zhang Q, Zhou J H, Zhang B and Wu E H. 2021. DsNet: dual stack network for detecting diabetes mellitus and chronic kidney disease. Information Sciences, 547: 945-962 [DOI: 10.1016/j.ins.2020.08.074]

Zhou M G, Wang H D, Zeng X Y, Yin P, Zhu J, Chen W Q, Li X H, Wang L J, Wang L M, Liu Y N, Liu J M, Zhang M, Qi J L, Yu S C, Afshin A, Gakidou E, Glenn S, Krish V S, Miller-Petrie M K, Mountjoy-Venning W C, Mullany E C, Redford S B, Liu H Y, Naghavi M, Hay S I, Wang L H, Murray C J L and Liang X F. 2019. Mortality, morbidity, and risk factors in China and its provinces, 1990-2017: a systematic analysis for the global burden of disease study 2017. Lancet, 394(10204): 1145-1158 [DOI: 10.1016/S0140-6736(19)30427-1]