浏览全部资源
扫码关注微信
纸质出版日期： 2022-03-16 ，
录用日期： 2021-09-06
移动端阅览
余力, 刘宵雪, 闫朝阳, 李建瑞, 张志强, 黄韫栀, 徐军. 面向多模态MRI脑胶质瘤区域三维分割与生存期预测的级联U-Net网络[J]. 中国图象图形学报, 2022,27(3):850-861.
Li Yu, Xiaoxue Liu, Chaoyang Yan, Jianrui Li, Zhiqiang Zhang, Yunzhi Huang, Jun Xu. Brain glioma tumor segmentation and survival prediction from multi-modality MRIs via cascaded U-Net[J]. Journal of Image and Graphics, 2022,27(3):850-861.
目的
2
针对常见方法对脑胶质瘤的肿瘤分割和生存预测需要单独建模的问题，提出一种带有变分自编码器(variational auto-encoder
VAE)分支的两阶段级联U-Net算法，旨在分割肿瘤的同时提取鲁棒的特征预测患者生存期，有助于患者的精准治疗。
方法
2
提出的两阶段级联U-Net网络，第1阶段实现初步粗分割，第2阶段实现精细化分割。此外，在第2阶段添加变分自编码器分支以提取更加鲁棒的特征并提高模型泛化性。其中，变分自编码器分支获取的特征被送入随机森林算法以进行生存期预测。另外，在两个阶段的解码器部分都添加了SE(squeeze-and-excitation)-残差模块以及注意力门模块，提高了分割精度。
结果
2
在Brain Tumor Segmentation (BraTS)竞赛官网分别评估了本文方法在BraTS2020验证集上分割以及总体生存期预测两大任务的结果，本文算法在该验证集的全肿瘤区域、肿瘤核心区域以及增强型肿瘤区域分别取得了90.66%、85.09%和79.02%的Dice相似系数。相较3DU-Net在3个肿瘤子区域的Dice相似系数分别提高了4.3%、1.37%和5.36%。对于总体生存期的预测准确率达到了55.2%，相较性能最佳算法的预测准确率虽然低3.4%，但是从均方误差这一评估指标来看略有优势。
结论
2
为了防止出现过拟合，提出带有VAE分支的两阶段级联U-Net网络结构，同时该分支能够提取肿瘤的鲁棒特征用于后续生存分析。实验表明，该网络能够较为准确地分割脑肿瘤，并且其提取的特征能够准确地预测患者生存期，可以为临床医生进行脑胶质瘤诊断提供有力的参考。
Objective
2
Glioma is the most common primary brain tumor with the highest mortality. The survival rate of different degrees of gliomas varies greatly. The magnetic resonance imaging based (MRI-based) scan imaging in the context of tumor substructure is composed of diversified intensity distributions
reflecting different tumor biological characteristics. Diagnosis grading and survival time prediction are conducted in terms of accurate detection and segmentation of abnormal tumor tissues. Radiomics aims to mine medical image data via a wide range of quantitative features extraction nowadays. The radiomics-based analyses are becoming essential for the medical prognosis. There is some constraints in designing and extracting features for traditional image omics. Deep learning methods are subjected of manual designing and extracting features. Deep learning convolutional neural networks (CNNs) has its priority in U-Net based image segmentation currently. Due to the time-consuming and labor-intensive labeling of gliomas and the lack of a large amount of supervision information
a U-Net based variational-auto-encoder (VAE) branch and the self-supervision information can deal with the over-fitting of the model effectively. The VAE branch can extract the potential robust features of images to resolve traditional image omics. A two-stage cascaded U-Net algorithm is demonstrated in context of VAE branch
which can realize the automatic segmentation of gliomas for diagnosis grading and the extraction of robust features derived from subsequent survival time prediction.
Method
2
In the first stage
the network predicts the coarse segmentation results in terms of the input of multi-modal MR images. The initial results and the multi-modal MR images are integrated as the input of the second stage for the network to improve the segmentation results. Moreover
a VAE branch is melted in the second stage of the network to obtain robust representative features and improve the generalization of the model. The representative features are used to predict the human life span based on the random forest algorithm. The traditional convolution gives the same weights to all channels of the input feature maps by default for feature fusion
while the squeeze-and-excitation (SE) module can learn the importance of different channels
so the SE module is evolved to the basic unit of this demonstrated algorithm. Furthermore
the prediction illustrates large differences in shape and size between patients
the cascaded network that detects and then segment can lead to the redundancy of model parameters. The proposed network adds an attention gate mechanism to the additive jump connection of the traditional U-Net
which achieves the same effect as the method that detects and then segment without increasing the amount of parameters. In addition
when the enhanced tumor area is small
this algorithm analysis is probably mislead the necrotic area and the non-enhanced tumor area as the enhanced tumor area. When the predicted enhanced tumor area is less than 500 voxels
the enhanced tumor area is substituted with necrotic and non-enhancing tumor area. This research method illustrates a random forest-based regressor for the prediction of the overall survival time and a random forest-based classifier for the prediction of category of the overall survival time at the end. The premise requires the use of low-dimensional potential features extracted by VAE and the clinical features of the patient's age
and feature option is implemented via the recursive feature elimination method based on random forest regression. When the prediction result of the regression model is inconsistent with the prediction category of the classification model
the final prediction result is divided into a fixed overall survival time according to the prediction category if the probability predicting the category exceeds 80% The short-term corresponds to 10 months
the mid-term corresponds to 12.5 months
and the long-term corresponds to 15 months.
Result
2
Our research analysis implements a five-fold cross validation on BraTS2020 training set. The generated result is the average of 5 validation results. The method achieves Dice similarity coefficients of 90.66%
85.09%
and 79.02% in the whole tumor area
tumor core area and enhanced tumor area in the validation set
respectively. In terms of the clinical feature of patient's age
the overall survival prediction accuracy rate reaches 41.4%
and the potential extracted features in the VAE branch
the accuracy rate could reach 55.2%. Although the accuracy rate is 3.5% lower than that of the first-ranked method
this proposed method has a slight advantage in terms of the mean square error.
Conclusion
2
the research proposes a two-stage cascaded U-Net network with a VAE branch to resolve over-fitting issue. Simultaneously
the branch can also extract the potential features of the tumor to predict the survival time. The test results demonstrate that the network can segment brain tumors more accurately
and the potential extracted features can predict suitable patient survival time
which can provide a powerful reference for clinicians in the diagnosis of gliomas.
脑胶质瘤3维自动分割生存期预测级联神经网络变分自编码器注意力机制
automatic 3D segmentation of gliomasurvival predictioncascade neural networkvariational auto-encoder(VAE)attention mechanism
Aerts H J W A, Velazquez E R, Leijenaar R T H, Parmar C, Grossmann P, Carvalho S, Bussink J, Monshouwer R, Haibe-Kains B, Rietveld D, Hoebers F, Rietbergen M M, Leemans C R, Dekker A, Quackenbush J, Gillies R J and Lambin P. 2014. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature Communications, 5: #4006[DOI: 10.1038/ncomms5006]
Agravat R R and Raval M S. 2019. Brain tumor segmentation and survival prediction//Proceedings of the 5th International MICCAI Brainlesion Workshop. Shenzhen, China: Springer: 338-348[DOI: 10.1007/978-3-030-46640-4_32http://dx.doi.org/10.1007/978-3-030-46640-4_32]
ÇiçekÖ, Abdulkadir A, Lienkamp S S, Brox T and Ronneberger O. 2016. 3D U-Net: learning dense volumetric segmentation from sparse annotation//Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention. Athens, Greece: Springer: 424-432[DOI: 10.1007/978-3-319-46723-8_49http://dx.doi.org/10.1007/978-3-319-46723-8_49]
Clough J R, Oksuz I, Puyol-Antón E, Ruijsink B, King A P and Schnabel J A. 2019. Global and local interpretability for cardiac MRI classification//Proceedings of the 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen, China: Springer: 656-664[DOI: 10.1007/978-3-030-32251-9_72http://dx.doi.org/10.1007/978-3-030-32251-9_72]
Feng X, Dou Q, Tustison N and Meyer C. 2019. Brain tumor segmentation with uncertainty estimation and overall survival prediction//Proceedings of the 5th International MICCAI Brainlesion Workshop. Shenzhen, China: Springer: 304-314[DOI: 10.1007/978-3-030-46640-4_29http://dx.doi.org/10.1007/978-3-030-46640-4_29]
Glas M, Happold C, Rieger J, Wiewrodt D, Bähr O, Steinbach J P, Wick W, Kortmann R D, Reifenberger G, Weller M and Herrlinger U. 2009. Long-term survival of patients with glioblastoma treated with radiotherapy and lomustine plus temozolomide. Journal of Clinical Oncology, 27(8): 1257-1261[DOI: 10.1200/JCO.2008.19.2195]
Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin P M and Larochelle H. 2017. Brain tumor segmentation with deep neural networks. Medical Image Analysis, 35: 18-31[DOI: 10.1016/j.media.2016.05.004]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Identity mappings in deep residual networks//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 630-645[DOI: 10.1007/978-3-319-46493-0_38http://dx.doi.org/10.1007/978-3-319-46493-0_38]
Hermoza R, Maicas G, Nascimento J C and Carneiro G. 2021. Post-hoc overall survival time prediction from brain MRI//Proceedings of the 18th IEEE International Symposium on Biomedical Imaging. Nice, France: IEEE: 1476-1480[DOI: 10.1109/ISBI48211.2021.9433877http://dx.doi.org/10.1109/ISBI48211.2021.9433877]
Hu J, Shen L and Sun G. 2018. Squeeze-and-excitation networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, USA: IEEE: 7132-7141[DOI: 10.1109/CVPR.2018.00745http://dx.doi.org/10.1109/CVPR.2018.00745]
Jiang Z K, Lyu X G, Zhang J X, Zhang Q and Wei X P. 2020, Review of deep learning methods for MRI brain tumor image segmentation. Journal of Image and Graphics, 25(2): 217-230
江宗康, 吕晓钢, 张建新, 张强, 魏小鹏. 2020. MRI脑肿瘤图像分割的深度学习方法综述. 中国图象图形学报, 25(2): 217-230 [DOI: 10.11834/jig.190173]
Jiang Z Y, Ding C X, Liu M F and Tao D C. 2019. Two-stage cascaded U-Net: 1st place solution to braTS challenge 2019 segmentation task//Proceedings of the 5th International MICCAI Brainlesion Workshop. Shenzhen, China: Springer: 231-241[DOI: 10.1007/978-3-030-46640-4_22http://dx.doi.org/10.1007/978-3-030-46640-4_22]
Kamnitsas K, Ledig C, Newcombe V F J, Simpson J P, Kane A D, Menon D K, Rueckert D and Glocker B. 2017. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis, 36: 61-78[DOI: 10.1016/j.media.2016.10.004]
Kaur T, Saini B S and Gupta S. 2019. An adaptive fuzzy K-nearest neighbor approach for MR brain tumor image classification using parameter free bat optimization algorithm. Multimedia Tools and Applications, 78(15): 21853-21890[DOI: 10.1007/s11042-019-7498-3]
Kofler F, Paetzold J C, Ezhov I, Shit S, Krahulec D, Kirschke J S, Zimmer C, Wiestler B and Menze B H. 2019. A baseline for predicting glioblastoma patient survival time with classical statistical models and primitive features ignoring image information//Proceedings of the 5th International MICCAI Brainlesion Workshop. Shenzhen, China: Springer: 254-261[DOI: 10.1007/978-3-030-46640-4_24http://dx.doi.org/10.1007/978-3-030-46640-4_24]
Leach M O, Brindle K M, Evelhoch J L, Griffiths J R, Horsman M R, Jackson A, Jayson G C, Judson I R, Knopp M V, Maxwell R J, McIntyre D, Padhani A R, Price P, Rathbone R, Rustin G J, Tofts P S, Tozer G M, Vennart W, Waterton J C, Williams S R and Workman P. 2005. The assessment of antiangiogenic and antivascular therapies in early-stage clinical trials using magnetic resonance imaging: issues and recommendations. British Journal of Cancer, 92(9): 1599-1610[DOI: 10.1038/sj.bjc.6602550]
Li Q, Bai K X, Zhao L and Guan X. 2020, Progresss and challenges of MRI brain tumor image segmentation. Journal of Image and Graphics, 25(3): 419-431
李锵, 白柯鑫, 赵柳, 关欣. 2020. MRI脑肿瘤图像分割研究进展及挑战. 中国图象图形学报, 25(3): 419-431 [DOI: 10.11834/jig.190524]
Louis D N, Perry A, Reifenberger G, Von Deimling A, Figarella-Branger D, Cavenee W K, Ohgaki H, Wiestler O D, Kleihues P and Ellison D W. 2016. The 2016 World Health Organization classification of tumors of the central nervous system: a summary. Acta Neuropathologica, 131(6): 803-820[DOI: 10.1007/s00401-016-1545-1]
Myronenko A. 2019. 3D MRI brain tumor segmentation using autoencoder regularization//Proceedings of the 4th International MICCAI Brainlesion Workshop. Granada, Spain: Springer: 311-320[DOI: 10.1007/978-3-030-11726-9_28http://dx.doi.org/10.1007/978-3-030-11726-9_28]
Oktay O, Schlemper J, Le Folgoc L, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla N Y, Kainz B, Glocker B and Rueckert D. 2018. Attention u-net: learning where to look for the pancreas[EB/OL]. [2020-06-10].https://arxiv.org/pdf/1804.03999.pdfhttps://arxiv.org/pdf/1804.03999.pdf
Patel A P, Tirosh I, Trombetta J J, Shalek A K, Gillespie S M, Wakimoto H, Cahill D P, Nahed B V, Curry W T, Martuza R L, Louis D N, Rozenblatt-Rosen O, Suvà M L, Regev A and Bernstein B E. 2014. Single-cell RNA-seq highlights intratumoral heterogeneity in primary glioblastoma. Science, 344(6190): 1396-1401[DOI: 10.1126/science.1254257]
Reza S M S, Samad M D, Shboul Z A, Jones K A and Iftekharuddin K M. 2019. Glioma grading using structural magnetic resonance imaging and molecular data. Journal of Medical Imaging, 6(2): #024501. [DOI: 10.1117/1.JMI.6.2.024501]
Saunders C N, Cornish A J, Kinnersley B, Law P J, Houlston R S and Collaborators. 2021. Searching for causal relationships of glioma: a phenome-wide Mendelian randomisation study. British Journal of Cancer, 124(2): 447-454[DOI: 10.1038/s41416-020-01083-1]
Tian X, Wang L and Ding Q. 2019, A review of image semantic segmentation methods based on deep learning. Journal of Software, 30(2): 440-468
田萱, 王亮, 丁琪. 2019. 基于深度学习的图像语义分割方法综述. 软件学报, 30(2): 440-468 [DOI: 10.13328/j.cnki.jos.005659]
Wang F F, Jiang R Z, Zheng L Q, Meng C and Biswal B. 2020. 3D U-Net based brain tumor segmentation and survival days prediction//Proceedings of the 5th International MICCAI Brainlesion Workshop. Shenzhen, China: Springer: 131-141[DOI: 10.1007/978-3-030-46640-4_13http://dx.doi.org/10.1007/978-3-030-46640-4_13]
Wang G T, Li W Q, Ourselin S and Vercauteren T. 2017. Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks//Proceedings of the 3rd International MICCAI Brainlesion Workshop. Quebec City, Canada: Springer: 178-190[DOI: 10.1007/978-3-319-75238-9_16http://dx.doi.org/10.1007/978-3-319-75238-9_16]
Weninger L, Rippel O, Koppers S and Merhof D. 2018. Segmentation of brain tumors and patient survival prediction: Methods for the brats 2018 challenge//Proceedings of the 4th International MICCAI Brainlesion Workshop. Granada, Spain: Springer: 3-12[DOI: 10.1007/978-3-030-11726-9_1http://dx.doi.org/10.1007/978-3-030-11726-9_1]
Zhang Q L, Zhao D, and Chi X B. 2017. A review of medical imaging diagnosis based on deep learning. Computer Science, 44(Z11): 1-7
张巧丽, 赵地, 迟学斌. 2017. 基于深度学习的医学影像诊断综述. 计算机科学, 44(Z11): 1-7 [DOI:10.11896/j.issn.1002-137X.2017.11A.001.]
Zhou C H, Ding C X, Lu Z T, Wang X C and Tao D C. 2018. One-pass multi-task convolutional neural networks for efficient brain tumor segmentation//Proceedings of the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention. Granada, Spain: Springer: 637-645[DOI: 10.1007/978-3-030-00931-1_73http://dx.doi.org/10.1007/978-3-030-00931-1_73]
相关作者
相关机构
京公网安备11010802024621