浏览全部资源
扫码关注微信
Received：10 September 2019，
Revised：15 May 2020，
Accepted：22 May 2020，
Published：16 February 2021
移动端阅览
利用深度学习方法对医学影像数据进行处理分析，极大地促进了精准医疗和个性化医疗的快速发展。深度学习在医学图像领域的应用较为广泛，具有多病种、多模态、多组学和多功能的特点。为便于对深度学习在医学图像处理领域的应用进行更深入有效的探索，本文系统综述了相关研究进展。首先，从深度学习在影像基因组学中的应用出发，理清了深度学习在医学影像领域应用的一般思路和现状，将医学影像领域分为智能诊断、疗效评估和预测预后等3个模块，并对模块内的各病种进行总结，展示了深度学习各算法的优缺点及面临的问题和挑战。其次，对深度学习中出现的新思路、新方法以及对传统方法的改进进行了阐述。最后，总结了该领域现阶段面临的问题，并对未来的研究方向做出了展望。基于深度学习的医学图像智能处理与分析虽然取得了一些有价值的研究成果，但还需要根据临床的实际需求，将深度学习与经典的机器学习算法及无创并且高效的多组学数据结合起来，对深度学习的理论和方法进行深入研究。
The amount of medical imaging data is increasing rapidly every year. Although large-scale medical imaging data pose considerable challenges to the work of clinicians
they also offer opportunities for improving disease diagnosis and treatment models. Algorithms based on deep learning exhibit advantages over humans in processing big data
analyzing complex and nondeterministic data
and delving into potential information that can be obtained from data. In recent years
an increasing number of scholars have use deep learning to process and analyze medical image data
promoting the rapid development of precision medicine and personalized medicine. The application of deep learning to medical image processing and analysis
which are characterized by multiple diseases
modals
functions
and omics
is relatively extensive. To facilitate the further exploration and effective application of deep learning methods by researchers in the field of medical image processing
this study systematically reviewed relevant research progress
expecting that such review will be beneficial for researchers in this field. First
general thoughts and the current situation of the application of deep learning to medical imaging were clarified from the perspective of deep learning applications to imaging genomics. Second
state-of-the-art ideas and methods and recent improvements in original deep learning methods were comprehensively described. Lastly
existing problems in this field were highlighted and development trends were explored. In accordance with application status
the application of deep learning to medical imaging was divided into three modules: intelligent diagnosis
response evaluation
and prediction prognosis. The modules were subdivided into different diseases for summary
and the advantages and disadvantages of each deep learning method and existing problems and challenges were highlighted. In terms of intelligent diagnosis
the disadvantages of manual doctor diagnosis
such as heavy workload
subjective cognitive susceptibility
low efficiency
and high misdiagnosis rate
are becoming increasingly evident due to the increasing complexity of medical imaging information. The use of deep learning to interpret medical images and then comparing the results with other case records will help doctors locate lesions and assist in diagnosis. Moreover
the burden of doctors and medical misjudgments can be effectively reduced
improving the accuracy of diagnosis and treatment. Further research on the applications of deep learning and computer vision technologies to radiography is a pressing task in the 21st century
particularly for diseases with high incidence
such as brain and fundus disorders. In the follow-up study
we should focus on optimizing the generation of labels
specifying precise pathological regions in medical images
and establishing a strong supervision model instead of a weak one. In addition
deploying a cropping algorithm on a picture archiving and communication system platform will pave the way to algorithm improvement and entry to the clinical environment. In terms of response evaluation
the pathological evaluation of surgical specimens is the only reliable indicator of long-term tumor prognosis. However
these pathological data can only be obtained after completing all preoperative and surgical treatments
and they cannot be used as a guide for adjusting treatment. The development of noninvasive biomarkers with early prediction potential is important. At present
most relevant studies have conducted analysis by using traditional machine learning algorithms or statistical methods. Biological and clinical data extracted using medical imaging artificial intelligence programs designed by precision medicine researchers can determine the level of lymphocyte infiltration into tumors
predict imaging omics indicators of the therapeutic effect of immunotherapy to patients
and guide chemoradiotherapy treatment. The realization and development of this technique are of considerable clinical significance and deserve additional effort from researchers. With regard to prediction prognosis
imaging markers can predict the mutation status of genes
the molecular categories that regulate the activity of treatment-related proteins
and disease status and prognosis by using deep learning. Intelligent processing and analysis of medical images using deep learning is noninvasive
repeatable
and inexpensive. In the succeeding research
the data fusion of different omics should be completed to realize a link model of the reasoning mechanism based on content and semantics. Moreover
a fast retrieval method for structured data should be established by using the correlation relationship among data to develop an intelligent prediction model with high accuracy and strong robustness. Valuable research results and meaningful progress of the intelligent processing and analysis of medical images based on deep learning have been obtained; however
they have not been widely used in the clinical setting. In-depth research on deep learning theories and methods should be conducted further. In particular
the acquisition of a large number of high-quality labeled imaging cases
multicenter research and verification
the visualization of the decision-making process and diagnosis basis
and the establishment of a tripartite evaluation system are critical. Moreover
the development of intelligent medical imaging requires the fusion of big data and medical imaging technologies
clinical experience and multiomics big data
and artificial intelligence and medical imaging capabilities. Medical problems and clinical results should be used as guides to realize micro/macro system precision micro-closed-loop research for solving practical clinical problems
such as accurate tumor segmentation before
during
and after surgery; intelligent disease diagnosis; and noninvasive tracking of treatment effect
treatment response
and disease status.
AubrevilleM, Knipfer C, Oetter N, Jaremenko C, Rodner E, Denzler J, Bohr C, Neumann H, Stelzle F and Maier A. 2017. Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Scientific Reports, 7(1): #11979[DOI: 10.1038/s41598-017-12320-8]
Bibault J E, Giraud P, Housset M, Durdux C, Taieb J, Berger A, Coriat R, Chaussade S, Dousset B, Nordlinger B and Burgun A. 2018. Deep learning and radiomics predict complete response after neo-adjuvant chemoradiation for locally advanced rectal cancer. Scientific Reports, 8(1): #12611[DOI: 10.1038/s41598-018-30657-6]
Chen R H, Li J J, Zhou X, Liu J J and Huang G. 2017b. Fructose-1, 6-Bisphosphatase 1 reduces 18 F FDG uptake in hepatocellular carcinoma. Radiology, 284(3): 844-853[DOI: 10.1148/radiol.2017161607]
Chen R H, Zhou X, Liu J J and Huang G. 2016. Relationship between 18 F-FDG PET/CT findings and HER2 expression in gastric cancer. Journal of Nuclear Medicine, 57(7): 1040-1044[DOI: 10.2967/jnumed.115.171165]
Chen R H, Zhou X, Liu J J and Huang G. 2019. Relationship between the expression of PD-1/PD-L1 and 18F-FDG uptake in bladder cancer. European Journal of Nuclear Medicine and Molecular Imaging, 46(4): 848-854[DOI: 10.1007/s00259-018-4208-8]
Chen Y P, Li J N, Xiao H X, Jin X J, Yan S C and Feng J S. 2017a. Dual path networks[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1707.01629.pdf https://arxiv.org/pdf/1707.01629.pdf
Choi H and Jin K H. 2018. Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. Behavioural Brain Research, 344: 103-109[DOI: 10.1016/j.bbr.2018.02.017]
Christopher M, Belghith A, Bowd C, Proudfoot J A, Goldbaum M H, Weinreb R N, Girkin C A, Liebmann J M and Zangwill L M. 2018. Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs. Scientific Reports, 8(1): #16685[DOI: 10.1038/s41598-018-35044-9]
Coroller T P, Agrawal V, Narayan V, Hou Y, Grossmann P, Lee S W, Mak R H and Aerts H J W L. 2016. Radiomic phenotype features predict pathological response in non-small cell lung cancer. Radiotherapy and Oncology, 119(3): 480-486[DOI: 10.1016/j.radonc.2016.04.004]
Coudray N, Ocampo P S, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, Moreira A L, Razavian N and Tsirigos A. 2018. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nature Medicine, 24(10): 1559-1567[DOI: 10.1038/s41591-018-0177-5]
Ding Y M, Sohn J H, Kawczynski M G, Trivedi H, Harnish R, Jenkins N W, Lituiev D, Copeland T P, Aboian M S, Aparici C M, Behr S C, Flavell R R, Huang S Y, Zalocusky K A, Nardo L, Seo Y, Hawkins R A, Pampaloni M H, Hadley D and Franc B L. 2019. A Deep learning model to predict a diagnosis of Alzheimer disease by using 18 F-FDG PET of the brain. Radiology, 290(2): 456-464[DOI: 10.1148/radiol.2018180958]
Eldan R and Shamir O. 2015. The power of depth for feedforward neural networks[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1512.03965.pdf https://arxiv.org/pdf/1512.03965.pdf
Esteva A, Kuprel B, Novoa R A, Ko J, Swetter S M, Blau H M and Thrun S. 2017. Correction: corrigendum: dermatologist-level classification of skin cancer with deep neural networks. Nature, 546(7660): #686[DOI: 10.1038/nature22985]
Fave X, Zhang L F, Yang J Z, Mackin D, Balter P, Gomez D, Followill D, Jones A K, Stingo F, Liao Z X, Mohan R and Court L. 2017. Delta-radiomics features for the prediction of patient outcomes in non-small cell lung cancer. Scientific Reports, 7(1): #588[DOI: 10.1038/s41598-017-00665-z]
Ferreira C A, Melo T, Sousa P, Meyer M I, Shakibapour E, Costa P and Campilho A. 2018. Classification of breast cancer histology images through transfer learning using a pre-trained inception resnet V2//Proceedings of the 15th International Conference Image Analysis and Recognition. Póvoa de Varzim, Portugal: Springer: 763-770[ DOI: 10.1007/978-3-319-93000-8_86 http://dx.doi.org/10.1007/978-3-319-93000-8_86 ]
Fukushima K. 1980. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36, 193-202[DOI:10.1007/BF00344251]
François C. 2016. Xception: deep learning with depthwise separable convolutions[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1610.02357.pdf https://arxiv.org/pdf/1610.02357.pdf
Gulshan V, Peng L, Coram M, Stumpe M C, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson P C, Mega J L and Webster D R. 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22): 2402-2410[DOI: 10.1001/jama.2016.17216]
Hao H X, Zhou Z G, Li S L, Maquilan G, Folkert M R, Iyengar P, Westover K D, Albuquerque K, Liu F, Choy H, Timmerman R, Yang L and Wang J. 2018. Shell feature: a new radiomics descriptor for predicting distant failure after radiotherapy in non-small cell lung cancer and cervix cancer. Physics in Medicine and Biology, 63(9): #095007[DOI: 10.1088/1361-6560/aabb5e]
Hosny A, Parmar C, Coroller T P, Grossmann P, Zeleznik R, Kumar A, Bussink J, Gillies R J, Mak R H and Aerts H J W L. 2018. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Medicine, 15(11): #e1002711[DOI: 10.1371/journal.pmed.1002711]
Hu J, Shen L, Samuel A, Sun G and Wu E H. 2020. Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8): 2011-2023[DOI: 10.1109/TPAMI.2019.2913372]
Huang G, Liu Z, van der Maaten L and Weinberger K. 2017. Densely connected convolutional networks[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1608.06993.pdf https://arxiv.org/pdf/1608.06993.pdf
Ioffe S and Szegedy C. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift//Proceedings of the 32nd International Conference on International Conference on Machine Learning. Lille, France: ACM: 448-456
Jun Y, Eo T, Kim T, Shin H, Hwang D, Bae S H, Park Y W, Lee H J, Choi B W and Ahn S S. 2018. Deep-learned 3D black-blood imaging using automatic labelling technique and 3D convolutional neural networks for detecting metastatic brain tumors. Scientific Reports, 8(1): #9450[DOI: 10.1038/s41598-018-27742-1]
Kermany D S, Goldbaum M, Cai W J, ValentimC C S, Liang H Y, Baxter S L, McKeown A, Yang G, Wu X K, Yan F B, Dong J, Prasadha M K, Pei J, Ting M Y L, Zhu J, Li C, Hewett S, Dong J S, Ziyar I, Shi A, Zhang R Z, Zheng L H, Hou R, Shi W, Fu X, Duan Y O, Huu V A N, Wen C, Zhang E D, Zhang C L, Li O L, Wang X B, Singer M A, Sun X D, Xu J, Tafreshi A, Lewis M A, Xia H M and Zhang K. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 172(5): 1122-1131.e9[DOI: 10.1016/j.cell.2018.02.010]
Krizhevsky A, Sutskever I and Hinton G E. 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84-90[DOI: 10.1145/3065386]
Kusumoto D, Lachmann M, Kunihiro T, Yuasa S, Kishino Y, Kimura M, Katsuki T, Itoh S, Seki T and Fukuda K. 2018. Automated deep learning-based system to identify endothelial cells derived from induced pluripotent stem cells. Stem Cell Reports, 10(6): 1687-1695[DOI: 10.1016/j.stemcr.2018.04.007]
He K M, Zhang X Y, Ren S Q and Sun J. 2015. Deep residual learning for image recognition[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1512.03385.pdf https://arxiv.org/pdf/1512.03385.pdf
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Identity mappings in deep residual networks[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1603.05027.pdf https://arxiv.org/pdf/1603.05027.pdf
Lao J W, Chen Y S, Li Z C, Li Q H, Zhang J, Liu J and Zhai G T. 2017. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Scientific Reports, 7(1): #10353[DOI: 10.1038/s41598-017-10649-8]
Lee C S, Baughman D M and Lee A Y. 2017. Deep learning is effective for classifying normal versus age-related macular degeneration OCT images. Ophthalmology Retina, 1(4): 322-327[DOI: 10.1016/j.oret.2016.12.009]
Lee H, Yune S, Mansouri M, Kim M, Tajmir S H, Guerrier C E, Ebert S A, Pomerantz S R, Romero J M, Kamalian S, Gonzalez R G, Lev M H and Do S. 2019. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nature Biomedical Engineering, 3(3): 173-182[DOI: 10.1038/s41551-018-0324-9]
Li Z J, Wang Y Y, Yu J H, Guo Y and Cao W. 2017. Deep learning based radiomics (DLR) and its usage in noninvasive IDH1 prediction for low grade glioma. Scientific Reports, 7(1): #5467[DOI: 10.1038/s41598-017-05848-2]
Limkin E J, Sun R, Dercle L, Zacharaki E I, Robert C, Reuzé S, Schernberg A, Paragios N, Deutsch E and Ferté C. 2017. Promises and challenges for the implementation of computational medical imaging (radiomics) in oncology. Annals of Oncology, 28(6): 1191-1206[DOI: 10.1093/annonc/mdx034]
Lin L, Dou Q, Jin Y M, Zhou G Q, Tang Y Q, Chen W L, Su B A, Liu F, Tao C J, Jiang N, Li J Y, Tang L L, Xie C M, Huang S M, Ma J, Heng P A, Wee J T S, Chua M L K, Chen H and Sun Y. 2019. Deep learningfor automated contouring of primary tumor volumes by MRI for nasopharyngeal carcinoma. Radiology, 291(3): 677-686[DOI: 10.1148/radiol.2019182012]
Lin M, Chen Q and Yan S C. 2013. Network in network[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1312.4400v3.pdf https://arxiv.org/pdf/1312.4400v3.pdf
LeCun Y, Bottou L, Bengio Y and Haffner P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324
Nie K, Shi L M, Chen Q, Hu X, Jabbour S K, Yue N, Niu T Y and Sun X N. 2016. Rectal cancer: assessment of neoadjuvant chemoradiation outcome based on radiomics of multiparametric MRI. Clinical Cancer Research, 22(21): 5256-5264[DOI: 10.1158/1078-0432.CCR-15-2997]
Oakden-Rayner L, Carneiro G, Bessen T, Nascimento J C, Bradley A P and Palmer L J. 2017. Precision radiology: predicting longevity using feature engineering and deep learning methods in a radiomics framework. Scientific Reports, 7(1): #1648[DOI: 10.1038/s41598-017-01931-w]
Rosenblatt F. 1957. The perceptron—a perceiving and recognizing automation. 85-460-1. Cornell Aeronautical Laboratory
Rios Velazquez E, Parmar C, Liu Y, Coroller T P, Cruz G, Stringfield O, Ye Z X, Makrigiorgos M, Fennessy F, Mak R H, Gillies R, Quackenbush J and Aerts H J W L. 2017. Somatic mutations drive distinct imaging phenotypes in lung cancer. Cancer Research, 77(14): 3922-3930[DOI: 10.1158/0008-5472.CAN-17-0122]
Schlegl T, Waldstein S M, Bogunovic H, Endstraβer F, Sadeghipour A, Philip A M, Podkowinski D, Gerendas B S, Langs G and Schmidt-Erfurth U. 2018. Fully automated detection and quantification of macular fluid in OCT using deep learning. Ophthalmology, 125(4): 549-558[DOI: 10.1016/j.ophtha.2017.10.031]
Shi J, Zheng X, Li Y, Zhang Q and Ying S H. 2018. Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of Alzheimer's disease. IEEE Journal of Biomedical and Health Informatics, 22(1): 173-183[DOI: 10.1109/JBHI.2017.2655720]
Sun R, Limkin E J, Vakalopoulou M, Dercle L, Champiat S, Han S R, Verlingue L, Brandao D, Lancia A, Ammari S, Hollebecque A, Scoazec J Y, Marabelle A, Massard C, Soria J C, Robert C, Paragios N, Deutsch E and Ferté C. 2018. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. The Lancet Oncology, 19(9): 1180-1191[DOI: 10.1016/S1470-2045(18)30413-3]
Szegedy C, Ioffe S, Vanhoucke V and Alemi A A. 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning//Proceedings of the 31st AAAI Conference on Artificial Intelligence. San Francisco, USA: AAAI: 4278-4284
Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V and Rabinovich A. 2015. Going deeper with convolutions//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE: 1-9[ DOI: 10.1109/CVPR.2015.7298594 http://dx.doi.org/10.1109/CVPR.2015.7298594 ]
Szegedy C, Vanhoucke V, Ioffe S, Shlens J and Wojna Z. 2016. Rethinking the inception architecture for computer vision//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 2818-2826[ DOI: 10.1109/CVPR.2016.308 http://dx.doi.org/10.1109/CVPR.2016.308 ]
Simonyan K and Zisserman A. 2014. Very deep convolutional networksfor large-scale image recognition[EB/OL].[2019-08-20] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf
Tang Y C. 2015. Deep learning using linear support vector machines[EB/OL].[2019-05-20] . https://arxiv.org/pdf/1306.0239.pdf https://arxiv.org/pdf/1306.0239.pdf
Ting D S W, Cheung C Y L, Lim G, Tan G S W, Quang N D, Gan A, Hamzah H, Garcia-Franco R, Yeo I Y S, Lee S Y, Wong E Y M, Sabanayagam C, Baskaran M, Ibrahim F, Tan N C, Finkelstein E A, Lamoureux E L, Wong I Y, Bressler N M, Sivaprasad S, Varma R, Jonas J B, He M G, Cheng C Y, Cheung G C M, Aung T, Hsu W, Lee M L and Wong T Y. 2017. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA, 318(22): 2211-2223[DOI: 10.1001/jama.2017.18152]
Titano J J, Badgeley M, Schefflein J, Pain M, Su A, Cai M, Swinburne N, Zech J, Kim J, Bederson J, Mocco J, Drayer B, Lehar J, Cho S, Costa A and Oermann E K. 2018. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nature Medicine, 24(9): 1337-1341[DOI: 10.1038/s41591-018-0147-y]
Vardhanabhuti V and Kuo M D. 2018. Lung cancer radiogenomics: the increasing value of imaging in personalized management of lung cancer patients. Journal of Thoracic Imaging, 33(1): 17-25[DOI: 10.1097/RTI.0000000000000312]
Wang S, Shi J Y, Ye Z X, Dong D, Yu D D, Zhou M, Liu Y, Gevaert O, Wang K, Zhu Y B, Zhou H Y, Liu Z Y and Tian J. 2019. Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. European Respiratory Journal, 53(3): #1800986[DOI: 10.1183/13993003.00986-2018]
Xie S N, Girshick R, Dollár P, Tu Z W and He K M. 2017. Aggregated residual transformations for deep neural networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 5987-5995[ DOI: 10.1109/CVPR.2017.634 http://dx.doi.org/10.1109/CVPR.2017.634 ]
Xie Y T, Xia Y, Zhang J P, Song Y, Feng D G, Fulham M and Cai W D. 2019. Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT. IEEE Transactions on Medical Imaging, 38(4): 991-1004[DOI: 10.1109/TMI.2018.2876510]
Ypsilantis P P, Siddique M, Sohn H M, Davies A, Cook G, Goh V and Montana G. 2015. Predicting response to neoadjuvant chemotherapy with PET Imaging using convolutional neural networks. PLoS One, 10(9): #e0137036[DOI: 10. 1371/journal. pone. 0137036]
Yu Y, Wang J H, Ng C W, Ma Y K, Mo S P, Fong E L S, Xing J W, Song Z W, Xie Y F, Si K, Wee A, Welsch R E, So P T C and Yu H. 2018. Deep learning enables automated scoring of liver fibrosis stages. Scientific Reports, 8(1): #16016[DOI: 10.1038/s41598-018-34300-2]
相关作者
相关机构
京公网安备11010802024621