浏览全部资源
扫码关注微信
Published： 16 July 2023 ，
移动端阅览
丁熠， 郑伟， 耿技， 邱泸谊， 秦志光. 2023. 基于多层级并行神经网络的多模态脑肿瘤图像分割框架. 中国图象图形学报， 28(07):2182-2194
Ding Yi， Zheng Wei， Geng Ji， Qiu Luyi， Qin Zhiguang. 2023. Multi-level parallel neural networks based multimodal human brain tumor image segmentation framework. Journal of Image and Graphics， 28(07):2182-2194
目的
2
在脑肿瘤临床诊疗过程中，由于医疗资源稀缺与诊断效率偏低，迫切需要高精度的医学图像分割工具进行辅助诊疗。目前，使用卷积神经网络进行脑肿瘤图像分割已经成为主流，但是其对于脑肿瘤信息的利用并不充分，导致精度与效率并不完善，而且重新设计一个全新且高效的深度神经网络模型是一项成本高昂的任务。为了更有效提取脑肿瘤图像中的特征信息，提出了基于多层级并行神经网络的多模态脑肿瘤图像分割框架。
方法
2
该框架基于现有的网络结构进行拓展，以ResNet（residual network）网络为基干，通过设计多层级并行特征提取模块与多层级并行上采样模块，对脑肿瘤的特征信息进行高效提取与自适应融合，增强特征信息的提取与表达能力。另外，受U-Net长连接结构的启发，在网络中加入多层级金字塔长连接模块，用于输入的不同尺寸特征之间的融合，提升特征信息的传播效率。
结果
2
实验在脑肿瘤数据集BRATS2015（brain tumor segmentation 2015）和BRATS2018（brain tumor segmentation 2018）上进行。在BRATS2015数据集中，脑肿瘤整体区、核心区和增强区的平均Dice值分别为84%、70%和60%，并且分割时间为5 s以内，在分割精度和时间方面都超过了当前主流的分割框架。在BRATS2018数据集中，脑肿瘤整体区、核心区和增强区的平均Dice值分别为87%、76%和71%，对比基干方法分别提高8%、7%和6%。
结论
2
本文提出多层级并行的多模态脑肿瘤分割框架，通过在脑肿瘤数据集上的实验验证了分割框架的性能，与当前主流的脑肿瘤分割方法相比，本文方法可以有效提高脑肿瘤分割的精度并缩短分割时间，对计算机辅助诊疗有重要意义。
Objective
2
Clinical-oriented magnetic resonance imaging （MRI） has its priority to analyze human brain tumors in relevance to such fields of MRI brain images of Flair， T1， T1c， and T2 modalities. The glioma can be oriented as the most common type of brain tumor for adults nowadays. Due to the features of its anatomical structure， such of lesions can be visually clarified in terms of MRI images analysis. The difference between medical treatment and scarcity of medical expertise are required of computers-assisted diagnosis and treatment. Recent deep learning technology for brain tumor segmentation has demonstrated its great potential for brain tumor image segmentation. In addition， to improve the segmentation accuracy of brain tumor images further， current literature reviews are focused on optimization of network feature extraction ability via an extraordinary network structure， and brain tumor images information are related to its multi-resolution information， spatial multi-view information， information post-processing， and symmetry information. Various of deep neural network （DNN） models have been developing in computer vision in recent years，such as Visual Geometry Group Network （VGGNet）， GoogLeNet， ResNet， and DenseNet. The DNN model mentioned above can facilitate the development of deep learning-based brain tumor diagnosis methods. To extract feature information in brain tumor images more effectively， we develop a multi-level parallel neural networks based multimodal brain tumor image segmentation framework further.
Method
2
To enhance the ability of feature information extraction and expression， this framework is facilitated derived from the existing network backbone， and the feature information of brain tumors can be extracted and fused adaptively via a multi-level parallel feature extraction module and parallel up-sampling module. Deeper features can be extracted in the depth of the network and iterate multiple backbone network branches for feature extraction in parallel. The layer-by-level connection of the neural network can not only broaden the width of the neural network but also mine the depth of the neural network. As a result， multi-level parallel feature extraction structure has its stronger and richer nonlinear representation capabilities than the single-level feature extraction structure， and more complex mapping transformations are fit into more complex image features as well. To preserve the richness of features， hierarchical parallel feature extraction structure has sufficient network width to extract various attribute information of images， such as different colors， shapes， spatial relationships， textures， and other related features. Furthermore， inspired by long connected structure of U-Net， a multi-level pyramid long connection module is melted into the network to fully achieve the integration of the input features of different sizes and improve the transmission efficiency of feature information. The richness of the feature is enhanced in terms of multi-level pyramid long connection module. Meanwhile， the input end of the multi-level pyramid long connection module can be used to fully analyze the information fusion between layers of different sizes. All of them can alleviate the loss and deformation of image information to a certain extent， which can improve the propagation efficiency of features of the same size at both ends of a long connection. It can affect the segmentation accuracy of multimodal brain tumor images ultimately.
Result
2
To verify the overall performance of the algorithm， an evaluation is first carried out on the testing set of the public brain tumor dataset BraTS2015. The average Dice scores of the proposed algorithm in the entire tumor， tumor core， and enhanced tumor areas can be reached to 84%， 70%， and 60% of each. It can optimize segmentation duration to less than 5 s farther. Some other related comparative experiments are linked to such modules of feature extraction， up-sampling， and the pyramid long connection， and the effectiveness of each module is compared with the backbone method as well. An experiment is conducted on the BraTS2018 validation set. The proposed algorithm can achieve average Dice scores of 87%， 76%， and 71% of each. Compared to the backbone method，it illustrates higher average Dice scores of 8.0%，7.0%， and 6.0%.
Conclusion
2
We extend the common network backbone and propose a multimodal brain tumor image segmentation framework based on a multi-level parallel neural network. We develop a multi-level parallel expansion at the same time. The hierarchical pyramid long connection module can be used to optimize original long connection model-derived multi-scale and receptive field-relevant unclear information， and the richness of features can be improved as well. Multi-level parallel neural network-based segmentation framework is demonstrated to optimize segmentation accuracy and efficiency.
多模态脑肿瘤图像多层级并行深度神经网络（DNN）特征融合语义分割
multimodal brain tumor imagemulti-level parallelismdeep neural network（DNN）feature fusionsemantic segmentation
Badrinarayanan V， Kendall A and Cipolla R. 2017. SegNet： a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence， 39（12）： 2481-2495 ［DOI： 10.1109/TPAMI.2016.2644615http://dx.doi.org/10.1109/TPAMI.2016.2644615］
Bakas S， Reyes M， Jakab A， Bauer B and Pawar K. Identifying the best machine learning algorithms for brain tumor segmentation， progression assessment， and overall survival prediction in the BRATS challenge ［EB/OL］. ［2021-12-24］. https://arxiv.org/pdf/1811.02629.pdfhttps://arxiv.org/pdf/1811.02629.pdf
Benson E， Pound M P， French A P， Jackson A S and Pridmore T P. 2018. Deep hourglass for brain tumor segmentation//Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Granada， Spain： Springer： 419-428 ［DOI： 10.1007/978-3-030-11726-9_37http://dx.doi.org/10.1007/978-3-030-11726-9_37］
Chen H， Qin Z G， Ding Y， Tian L and Qin Z. 2020. Brain tumor segmentation with deep convolutional symmetric neural network. Neurocomputing， 392： 305-313 ［DOI： 10.1016/j.neucom.2019.01.111http://dx.doi.org/10.1016/j.neucom.2019.01.111］
Chen L C， Zhu Y K， Papandreou G， Schroff F and Adam H. 2018a. Encoder-decoder with Atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 833-851 ［DOI： 10.1007/978-3-030-01234-2_49http://dx.doi.org/10.1007/978-3-030-01234-2_49］
Chen X， Liew J H， Xiong W， Chui C K and Ong S H. 2018b. Focus， segment and erase： an efficient network for multi-label brain tumor segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 674-689 ［DOI： 10.1007/978-3-030-01261-8_40http://dx.doi.org/10.1007/978-3-030-01261-8_40］
Ding Y， Chen F J， Zhao Y， Wu Z X， Zhang C and Wu D Y. 2019a. A stacked multi-connection simple reducing net for brain tumor segmentation. IEEE Access， 7： 104011-104024 ［DOI： 10.1109/ACCESS.2019.2926448http://dx.doi.org/10.1109/ACCESS.2019.2926448］
Ding Y， Dong R F， Lan T， Li X R， Shen G Y， Chen H and Qin Z G. 2018. Multi-modal brain tumor image segmentation based on SDAE. International Journal of Imaging Systems and Technology， 28（1）： 38-47 ［DOI： 10.1002/ima.22254http://dx.doi.org/10.1002/ima.22254］
Ding Y， Gong L P， Zhang M F， Li C and Qin Z G. 2020. A multi-path adaptive fusion network for multimodal brain tumor segmentation. Neurocomputing， 412： 19-30 ［DOI： 10.1016/j.neucom.2020.06.078http://dx.doi.org/10.1016/j.neucom.2020.06.078］
Ding Y， Li C， Yang Q Q， Qin Z and Qin Z G. 2019b. How to improve the deep residual network to segment multi-modal brain tumor images. IEEE Access， 7： 152821-152831 ［DOI： 10.1109/ACCESS.2019.2948120http://dx.doi.org/10.1109/ACCESS.2019.2948120］
Ding Y， Luo C J， Li C， Lan T and Qin Z G. 2019c. High-order correlation detecting in features for diagnosis of Alzheimer’s disease and mild cognitive impairment. Biomedical Signal Processing and Control， 53： #101564 ［DOI： 10.1016/j.bspc.2019.101564http://dx.doi.org/10.1016/j.bspc.2019.101564］
Ding Y， Wu G Z， Chen D J， Zhang N， Gong L P， Cao M S and Qin Z G. 2021. DeepEDN： a deep-learning-based image encryption and decryption network for internet of medical things. IEEE Internet of Things Journal， 8（3）： 1504-1518 ［DOI： 10.1109/JIOT.2020.3012452http://dx.doi.org/10.1109/JIOT.2020.3012452］
Dvořák P and Menze B. 2015. Local structure prediction with convolutional neural networks for multimodal brain tumor segmentation//Proceedings of International Workshop on Medical Computer Vision： Algorithms for Big Data. Munich， Germany： Springer： 59-71 ［DOI： 10.1007/978-3-319-42016-5_6http://dx.doi.org/10.1007/978-3-319-42016-5_6］
El-Dahshan E S A， Mohsen H M， Revett K and Salem A B M. 2014. Computer-aided diagnosis of human brain tumor through MRI： a survey and a new algorithm. Expert Systems with Applications， 41（11）： 5526-5545 ［DOI： 10.1016/j.eswa.2014.01.021http://dx.doi.org/10.1016/j.eswa.2014.01.021］
Goetz M， Weber C， Binczyk F， Polanska J， Tarnawski R， Bobek-Billewicz B， Koethe U， Kleesiek J， Stieltjes B and Maier-Hein K H. 2016. DALSA： domain adaptation for supervised learning from sparsely annotated MR images. IEEE Transactions on Medical Imaging， 35（1）： 184-196 ［DOI： 10.1109/TMI.2015.2463078http://dx.doi.org/10.1109/TMI.2015.2463078］
Guo J， Deng J K， Xue N N and ZafeiriouS. 2018. Stacked dense U-nets with dual transformers for robust face alignment//Proceedings of British Machine Vision Conference 2018. Newcastle， UK： BMVC：1-13
Havaei M， Davy A， Warde-Farley D， Biard A， Courville A， Bengio Y， Pal C， Jodoin P M and Larochelle H. 2017. Brain tumor segmentation with deep neural networks. Medical Image Analysis， 35： 18-31 ［DOI： 10.1016/j.media.2016.05.004http://dx.doi.org/10.1016/j.media.2016.05.004］
Havaei M， Dutil F， Pal C， Larochelle H and Jodoin P M. 2015. A convolutional neural network approach to brain tumor segmentation//Proceedings of the 1st International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Munich， Germany： Springer： 195-208 ［DOI： 10.1007/978-3-319-30858-6_17http://dx.doi.org/10.1007/978-3-319-30858-6_17］
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Hu J， Shen L and Sun G. 2018. Squeeze-and-excitation networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 7132-7141 ［DOI： 10.1109/CVPR.2018.00745http://dx.doi.org/10.1109/CVPR.2018.00745］
Islam M and Ren H L. 2017. Fully convolutional network with hypercolumn features for brain tumor segmentation//Proceedings of the 3rd International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Quebec City， Canada： Springer： 298-308 ［DOI： 10.1007/978-3-319-75238-9_26http://dx.doi.org/10.1007/978-3-319-75238-9_26］
Kamnitsas K， Ledig C， Newcombe V F J， Simpson J P， Kane A D， Menon D K， Rueckert D and Glocker B. 2017. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis， 36： 61-78 ［DOI： 10.1016/j.media.2016.10.004http://dx.doi.org/10.1016/j.media.2016.10.004］
Krähenbühl P and Koltun V. 2011. Efficient inference in fully connected CRFs with Gaussian edge potentials//Proceedings of the 24th International Conference on Neural Information Processing Systems. Granada， Spain： Curran Associates Inc.： 109-117
Krizhevsky A， Sutskever I and Hinton G E. 2012. ImageNet classification with deep convolutional neural networks//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe， USA： Curran Associates Inc.： 1097-1105
Luo K K， Wang T and Ye F F. 2021. U-Net segmentation model of brain tumor MR image based on attention mechanism and multi-view fusion. Journal of Image and Graphics， 26（9）： 2208-2218
罗恺锴， 王婷， 叶芳芳. 2021. 引入注意力机制和多视角融合的脑肿瘤MR图像U-Net分割模型. 中国图象图形学报， 26（9）： 2208-2218 ［DOI： 10.11834/jig.200584http://dx.doi.org/10.11834/jig.200584］
Martínez-Murcia F J， Górriz J M， Ramírez J， Alvarez Illán I， Salas-González D and Segovia F. 2014. Projecting MRI brain images for the detection of Alzheimer’s disease//Proceedings of the 2nd KES International Conference on Innovation in Medicine and Healthcares. San Sebastian， Spain： IOS Press： 225-233 ［DOI： 10.3233/978-1-61499-474-9-225http://dx.doi.org/10.3233/978-1-61499-474-9-225］
Meng Z Y， Gu X， Liang Y C， Xu D and Wu C G. 2021. Deep neural architecture search： a survey. Journal of Computer Research and Development， 58（1）： 22-33
孟子尧， 谷雪， 梁艳春， 许东， 吴春国. 2021. 计算机研究与发展， 58（1）： 22-33 ［DOI： 10.7544/issn1000-1239.2021.20190851http://dx.doi.org/10.7544/issn1000-1239.2021.20190851］
Menze B H， Jakab A， Bauer S， Kalpathy-Cramer J， Farahani K， Kirby J， Burren Y， Porz N， Slotboom J， Wiest R， Lanczi L， Gerstner E， Weber M A， Arbel T， Avants B B， Ayache N， Buendia P， Collins D L， Cordier N， Corso J J， Criminisi A， Das T， Delingette H， Demiralp Ç， Durst C R， Dojat M， Doyle S， Festa J， Forbes F， Geremia E， Glocker B， Golland P， Guo X T， Hamamci A， Iftekharuddin K M， Jena R， John N M， Konukoglu E， Lashkari D， Mariz J A， Meier R， Pereira S， Precup D， Price S J， Raviv T R， Reza S M S， Ryan M， Sarikaya D， Schwartz L， Shin H C， Shotton J， Silva C A， Sousa N， Subbanna N K， Szekely G， Taylor T J， Thomas O M， Tustison N J， Unal G， Vasseur F， Wintermark M， Ye D H， Zhao L， Zhao B S， Zikic D， Prastawa M， Reyes M and Van Leemput K. 2015. The multimodal brain tumor image segmentation benchmark （BRATS）. IEEE Transactions on Medical Imaging， 34（10）： 1993-2024 ［DOI： 10.1109/TMI.2014.2377694http://dx.doi.org/10.1109/TMI.2014.2377694］
Pereira S， Pinto A， Alves V and Silva C A. 2015. Deep convolutional neural networks for the segmentation of gliomas in multi-sequence MRI//Proceedings of the 1st International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Munich， Germany： Springer： 131-143 ［DOI： 10.1007/978-3-319-30858-6_12http://dx.doi.org/10.1007/978-3-319-30858-6_12］
Pereira S， Pinto A， Amorim J， Ribeiro A， Alves V and Silva C A. 2019. Adaptive feature recombination and recalibration for semantic segmentation with fully convolutional networks. IEEE Transactions on Medical Imaging， 38（12）： 2914-2925 ［DOI： 10.1109/TMI.2019.2918096http://dx.doi.org/10.1109/TMI.2019.2918096］
Rezaei M， Harmuth K， Gierke W， Kellermeier T， Fischer M， Yang H J and Meinel C. 2017. A conditional adversarial network for semantic segmentation of brain tumor//Proceedings of the 3rd International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Quebec City， Canada： Springer： 241-252 ［DOI： 10.1007/978-3-319-75238-9_21http://dx.doi.org/10.1007/978-3-319-75238-9_21］
Rezaei M， Yang H J and Meinel C. 2018. Voxel-GAN： adversarial framework for learning imbalanced brain tumor segmentation//Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Granada， Spain： Springer： 321-333 ［DOI： 10.1007/978-3-030-11726-9_29http://dx.doi.org/10.1007/978-3-030-11726-9_29］
Ronneberger O， Fischer P and Brox T. 2015. U-net： convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich， Germany： Springer： 234-241 ［DOI： 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28］
Roth H R， Lu L， Seff A， Cherry K M， Hoffman J， Wang S J， Liu J M， Turkbey E and Summers R M. 2014. A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations//Proceedings of the 17th International Conference on Medical Image Computing and Computer-Assisted Intervention. Boston， USA： Springer： 520-527 ［DOI： 10.1007/978-3-319-10404-1_65http://dx.doi.org/10.1007/978-3-319-10404-1_65］
Salehi S S M， Erdogmus D and Gholipour A. 2017. Auto-context convolutional neural network （auto-net） for brain extraction in magnetic resonance imaging. IEEE Transactions on Medical Imaging， 36（11）： 2319-2330 ［DOI： 10.1109/TMI.2017.2721362http://dx.doi.org/10.1109/TMI.2017.2721362］
Shaikh M， Anand G， Acharya G， Amrutkar A， Alex V and Krishnamurthi G. 2017. Brain tumor segmentation using dense fully convolutional neural network//Proceedings of the 3rd International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Quebec City， Canada： Springer： 309-319 ［DOI： 10.1007/978-3-319-75238-9_27http://dx.doi.org/10.1007/978-3-319-75238-9_27］
Shang H B， Zhao W G and Zhang W F. 2012. Preoperative assessment using multimodal functional magnetic resonance imaging techniques in patients with brain gliomas. Turkish Neurosurgery， 22（5）： 558-565 ［DOI： 10.5137/1019-5149.JTN.5332-11.1http://dx.doi.org/10.5137/1019-5149.JTN.5332-11.1］
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition//Proceedings of the 3rd International Conference on Learning Representations. San Diego， USA： ICLR： 28-29 ［DOI： 10.48550/arXiv.1409.1556http://dx.doi.org/10.48550/arXiv.1409.1556］
Tan M X and Le Q V. 2019. EfficientNet： rethinking model scaling for convolutional neural networks//Proceedings of the 36th International Conference on Machine Learning. Long Beach， USA： ICML： 6105-6114
Tuan T A， Tuan T A and Bao P T. 2018. Brain tumor segmentation using bit-plane and UNET［C］//Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries. Granada， Spain： Springer： 466-475 ［DOI： 10.1007/978-3-030-11726-9_41http://dx.doi.org/10.1007/978-3-030-11726-9_41］
Tustison N J， Avants B B， Cook P A， Zheng Y J， Egan A， Yushkevich P A and Gee J C. 2010. N4ITK： improved N3 bias correction. IEEE Transactions on Medical Imaging， 29（6）： 1310-1320 ［DOI： 10.1109/TMI.2010.2046908http://dx.doi.org/10.1109/TMI.2010.2046908］
Xia F， Shao H J and Deng X. 2022. Cross-stage deep-learning-based MRI fused images of human brain tumor segmentation. Journal of Image and Graphics， 27（3）： 873-884
夏峰， 邵海见， 邓星. 2022. 融合跨阶段深度学习的脑肿瘤MRI图像分割. 中国图象图形学报， 27（3）： 873-884 ［DOI： 10.11834/jig.210330http://dx.doi.org/10.11834/jig.210330］
Yu K， Jia L， Chen Y Q and Xu W. 2013. Deep learning： yesterday， today， and tomorrow. Journal of Computer Research and Development， 50（9）： 1799-1804
余凯， 贾磊， 陈雨强， 徐伟. 2013. 深度学习的昨天、今天和明天. 计算机研究与发展， 50（9）： 1799-1804
Zhao X M， Wu Y H， Song G D， Li Z Y， Zhang Y Z and Fan Y. 2018. A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Medical Image Analysis， 43： 98-111 ［DOI： 10.1016/j.media.2017.10.002http://dx.doi.org/10.1016/j.media.2017.10.002］
Zhou Z X， He Z S and Jia Y Y. 2020. AFPNet： a 3D fully convolutional neural network with Atrous-convolution feature pyramid for brain tumor segmentation via MRI images. Neurocomputing， 402： 235-244 ［DOI： 10.1016/j.neucom.2020.03.097http://dx.doi.org/10.1016/j.neucom.2020.03.097］
Zikic D， Ioannou Y， Brown M and Criminisi A. 2014. Segmentation of brain tumor tissues with convolutional neural networks//Proceedings of 2014 MICCAI Workshop on Multimodal Brain Tumor Segmentation Challenge. Boston， USA： MICCAI： 36-39
相关文章
相关作者
相关机构
京公网安备11010802024621