浏览全部资源
扫码关注微信
纸质出版日期： 2024-09-16 ，
移动端阅览
张建新， 高程阳， 孙鉴， 丁雪妍， 刘斌. 2024. 混合监督学习的乳腺癌全切片病理图像分类. 中国图象图形学报， 29(09):2825-2836
Zhang Jianxin， Gao Chengyang， Sun Jian， Ding Xueyan， Liu Bin. 2024. Whole slide pathological image classification of breast cancer based on mixed supervision learning. Journal of Image and Graphics， 29(09):2825-2836
目的
2
自监督与弱监督学习是解决乳腺癌全切片病理图像分类标注困难的有效方式。然而，由于组织病理图像的复杂性与多样性，仅依靠自监督学习生成的伪标签可能无法准确反映图像真实类别信息；同时，单一弱监督学习方法又存在标签信息匮乏等问题，在病理图像学习过程中易受干扰而导致预测结果不稳定。为此，提出了一种混合监督学习的乳腺癌全切片病理图像分类方法。
方法
2
首先，使用基于MoBY自监督框架进行训练，通过对比学习方式深入挖掘乳腺癌病理图像内在结构信息；然后，采用弱监督多示例学习方法进一步优化自监督模型，来获得更精准的判别示例；最后，从每幅全切片中筛选出具有代表性的乳腺癌病理图像关键示例，并借助Transformer编码器实现关键示例的特征融合以增强不同病理图像块之间的关联性，从而实现乳腺癌全切片病理图像的高精度分类。
结果
2
在公开的Camelyon-16乳腺癌病理图像数据集上进行实验评估，相比于该数据集上既有最优弱监督和自监督方法，本文方法的曲线下面积值分别可提升2.34%和2.74%，验证了所提出混合监督学习方法的有效性。此外，在MSK（Memorial Sloan-Kettering）腺癌病理外部验证数据集上较有监督方法取得了6.26%的性能提升，表明了本文方法的良好泛化能力。
结论
2
提出了混合监督学习的乳腺癌全切片病理图像分类方法，通过集成MoBY自监督对比学习与Transformer弱监督多示例学习，实现了乳腺癌全切片病理图像的更准确分类。
Objective
2
Breast cancer belongs to the most common malignant tumors among women， and its early diagnosis and accurate classification bear great importance. Breast cancer whole slide pathological images serve as important auxiliary diagnostic means， and their classification can assist doctors in the accurate identification of tumor types. However， given the complexity and huge data volume of breast cancer whole slide pathological images， manual annotation of the label of each image becomes time consuming and labor intensive. Therefore， researchers have proposed various automated methods to address the issue encountered in the classification of breast cancer whole slide pathological images. Self- and weakly supervised learning effectively tackling the challenge of breast cancer whole slide pathological image classification. Self-supervised learning is a type of machine learning method that skips the manual annotation of labels. This method design tasks that enable the model to learn feature representations from unlabeled data. Self-supervised learning has achieved remarkable progress in the field of computer vision， but it still faces certain challenges in breast cancer whole slide pathological image classification. Given the complexity and diversity of pathological images， relying solely on the pseudo labels generated by self-supervised learning may fail to accurately reflect the true classification information， which affects the classification performance. On the other hand， weakly supervised learning leverages information from unlabeled image data through various methods， such as multiple instance learning or label propagation. However， the associated models encounter challenges， such as limited label information and noise， which affect the model’s stability during the learning process and thus the stability of prediction results. To overcome the limitations of self- and weakly supervised learning， this paper proposes a mixed supervised learning method for breast cancer whole slide pathological image classification. The integration of MoBY self-supervised contrastive learning with weakly supervised multi-instance learning combines the advantages of these learning architectures and makes full use of unlabeled and noisy labeled data. In addition， such combination improves the classification performance through feature selection and spatial correlation enhancement， which results in increased robustness.
Method
2
First， the self-supervised MoBY was used to train the model on unlabeled pathological image data. MoBY， can learn key feature representations from images， is a self-supervised learning method based on self-reconstruction and contrastive learning. This process enables the model to extract useful feature information from unlabeled data and provide better initialization parameters for subsequent classification tasks. Then， a weakly supervised learning approach based on multiple instance learning was used for further model optimization. Multiple instance learning utilizes information from unlabeled image data for model training. In breast cancer whole slide pathological image classification， the accurate annotation of each image category often presents a challenge. This type of learning divides images into positive and negative instances based on instance-level labels to train the model. This approach partially contributes to solving the problem of limited label information and improves a model’s robustness and generalization capability. For the feature selection stage， representative feature vectors were selected from each whole slide image to reduce redundancy and noise， extract the most informative features， and improve the model’s focus and discriminative capability toward key regions. In addition， the paper leverages a Transformer encoder to improve the correlation among various image patches. The Transformer encoder is a powerful tool for modeling global contextual information in images， and it captures semantic relationships between different regions of an image to further increase the classification accuracy. The introduction of the Transformer encoder into breast cancer whole slide pathological image classification enables the improved utilization of global image information and further understanding of a model’s image structure and context. Comprehensive application of methods， such as self- and weakly supervised learning， resulted in the high accuracy and robustness of the proposed mixed supervised learning approach for the classification of breast cancer whole slide pathological images in this paper. In experiments， this method achieved excellent classification results on a dataset of breast cancer whole slide pathological images. This approach serves as a powerful tool and technical support for the early diagnosis and accurate classification of breast cancer.
Result
2
The effectiveness of the mixed supervised model was validated through evaluation experiments conducted on the publicly available Camelyon-16 breast-cancer pathological image dataset. Compared with the state-of-the-art weakly and self-supervised models of this dataset， the proposed model achieved evident improvements of 2.34% and 2.74% in the area under the receiver operating characteristic， respectively. This finding indicates that the proposed method outperformed the other models in terms of breast cancer whole slide pathological image classification tasks. To further validate its generalization capability， we performed experiments on an external validation dataset of MSK. The proposed model for this validation dataset demonstrated a great performance improvement of 6.26%， which further confirms its strong generalization capability and practicality.
Conclusion
2
The proposed breast cancer whole slide pathological image classification method based on mixed supervision achieved remarkable results in addressing the related challenge By leveraging the advantages of self-supervised learning， weakly supervised learning， and spatial correlation enhancement， the given model demonstrated improved classification performance on public and external validation datasets. This method exhibits a good generalization capability and offers a viable solution for the early diagnosis and treatment of breast cancer. Future research should further refine and optimize the proposed method to increase its accuracy and robustness in breast cancer whole slide pathology image classification. This paper will address the challenges in breast cancer pathological image classification and contribute to the development of early breast cancer diagnosis and treatment.
乳腺癌全切片病理图像分类混合监督学习特征融合Transformer
breast cancer whole slide pathology imageclassificationmixed supervised learningfeature fusionTransformer
Anderson B O， Ilbawi A M， Fidarova E， Weiderpass E， Stevens L， Abdel-Wahab M and Mikkelsen B. 2021. The global breast cancer initiative： a strategic collaboration to strengthen health care for non-communicable diseases. The Lancet Oncology， 22（5）： 578-581 ［DOI： 10.1016/S1470-2045（21）00071-1http://dx.doi.org/10.1016/S1470-2045（21）00071-1］
Bejnordi B E， Veta M， van Diest P J， van Ginneken B， Karssemeijer N， Litjens G， van der Laak J A W M， Hermsen M， Manson Q F， Balkenhol M， Geessink O， Stathonikos N， van Dijk M C， Bult P， Beca F， Beck A H， Wang D Y， Khosla A， Gargeya R， Irshad H， Zhong A X， Dou Q， Li Q Z， Chen H， Lin H J， Heng P A， Haß C， Bruni E， Wong Q， Halici U， Öner M Ü， Cetin-Atalay R， Berseth M， Khvatkov V， Vylegzhanin A， Kraus O， Shaban M， Rajpoot N， Awan R， Sirinukunwattana K， Qaiser T， Tsang Y W， Tellez D， Annuscheit J， Hufnagl P， Valkonen M， Kartasalo K， Latonen L， Ruusuvuori P， Liimatainen K， Albarqouni S， Mungal B， George A， Demirci S， Navab N， Watanabe S， Seno S， Takenaka Y， Matsuda H， Ahmady Phoulady H， Kovalev V， Kalinovsky A， Liauchuk V， Bueno G， Fernandez-Carrobles M M， Serrano I， Deniz O， Racoceanu D and Venâncio R. 2017. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA， 318（22）： 2199-2210 ［DOI： 10.1001/jama.2017.14585http://dx.doi.org/10.1001/jama.2017.14585］
Campanella G， Hanna M G， Geneslaw L， Miraflor A， Silva V W K， Busam K J， Brogi E， Reuter V E， Klimstra D S and Fuchs T J. 2019. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine， 25（8）： 1301-1309 ［DOI： 10.1038/s41591-019-0508-1http://dx.doi.org/10.1038/s41591-019-0508-1］
Chen R J and Krishnan R G. 2022. Self-supervised vision Transformers learn visual concepts in histopathology ［EB/OL］. ［2023-05-20］. https://arxiv.org/pdf/2203.00585.pdfhttps://arxiv.org/pdf/2203.00585.pdf
Chen X L， Fan H Q， Girshick R B and He K M. 2020. Improved baselines with momentum contrastive learning ［EB/OL］. ［2023-05-20］. https://arxiv.org/pdf/2003.04297.pdfhttps://arxiv.org/pdf/2003.04297.pdf
Cheng C L， Azhar R， Sng S H A， Chua Y Q， Hwang J S G， Chin J P F， Seah W K， Loke J C L， Ang R H L and Tan P H. 2016. Enabling digital pathology in the diagnostic setting： navigating through the implementation journey in an academic medical centre. Journal of Clinical Pathology， 69（9）： 784-792 ［DOI： 10.1136/jclinpath-2015-203600http://dx.doi.org/10.1136/jclinpath-2015-203600］
Chikontwe P， Nam S J， Go H， Kim M， Sung H J and Park S H. 2022. Feature re-calibration based multiple instance learning for whole slide image classification//Proceedings of the 25th International Conference on Medical Image Computing and Computer Assisted Intervention. Singapore， Singapore： Springer： 420-430 ［DOI： 10.1007/978-3-031-16434-7_41http://dx.doi.org/10.1007/978-3-031-16434-7_41］
Dosovitskiy A， Beyer L， Kolesnikov A， Weissenborn D， Zhai X H， Unterthiner T， Dehghani M， Minderer M， Heigold G， Gelly S， Uszkoreit J and Houlsby N. 2021. An image is worth 16 × 16 words： Transformers for image recognition at scale ［EB/OL］. ［2023-05-20］. https://arxiv.org/pdf/2010.11929.pdfhttps://arxiv.org/pdf/2010.11929.pdf
Grill J B， Strub F， Altché F， Tallec C， Richemond P H， Buchatskaya E， Doersch C， Pires B A， Guo Z D， Azar M G， Piot B， Kavukcuoglu K， Munos R and Valko M. 2020. Bootstrap your own latent a new approach to self-supervised learning//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver， Canada： Curran Associates Inc.： 21271-21284
Guan Y H， Zhang J， Tian K， Yang S， Dong P， Xiang J X， Yang W， Huang J Z， Zhang Y Y and Han X. 2022. Node-aligned graph convolutional network for whole slide image representation and classification//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans， USA： IEEE： 18813-18823 ［DOI： 10.1109/CVPR52688.2022.01825http://dx.doi.org/10.1109/CVPR52688.2022.01825］
Jin X， Wen K， Lyu G F， Shi J， Chi M X， Wu Z and An H. 2020. Survey on the applications of deep learning to histopathology. Journal of Image and Graphics， 25（10）： 1982-1993
金旭， 文可， 吕国锋， 石军， 迟孟贤， 武铮， 安虹. 2020. 深度学习在组织病理学中的应用综述. 中国图象图形学报， 25（10）： 1982-1993 ［DOI： 10.11834/jig.200460http://dx.doi.org/10.11834/jig.200460］
Koohbanani N A， Unnikrishnan B， Khurram S A， Krishnaswamy P and Rajpoot N. 2021. Self-path： self-supervision for classification of pathology images with limited annotations. IEEE Transactions on Medical Imaging， 40（10）： 2845-2856 ［DOI： 10.1109/TMI.2021.3056023http://dx.doi.org/10.1109/TMI.2021.3056023］
Li B， Li Y and Eliceiri K W. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 14318-14328 ［DOI： 10.1109/cvpr46437.2021.01409http://dx.doi.org/10.1109/cvpr46437.2021.01409］
Liu Z， Lin Y T， Cao Y， Hu H， Wei Y X， Zhang Z， Lin S and Guo B N. 2021. Swin Transformer： hierarchical vision Transformer using shifted windows//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal， Canada： IEEE： 10012-10022. ［DOI： 10.1109/ICCV48922.2021.00986http://dx.doi.org/10.1109/ICCV48922.2021.00986］
Lu M Y， Williamson D F K， Chen T Y， Chen R J， Barbieri M and Mahmood F. 2021. Data-efficient and weakly supervised computational pathology on whole slide images. Nature Biomedical Engineering， 5（6）： 555-570 ［DOI： 10.1038/s41551-020-00682-whttp://dx.doi.org/10.1038/s41551-020-00682-w］
Paszke A， Gross S， Massa F， Lerer A， Bradbury J， Chanan G， Killeen T， Lin Z M， Gimelshein N， Antiga L， Desmaison A， Köpf A， Yang E， DeVito Z， Raison M， Tejani A， Chilamkurthy S， Steiner B， Fang L， Bai J J and Chintala S. 2019. PyTorch： an imperative style， high-performance deep learning library//Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver， Canada： Curran Associates Inc.： 8026-8037
Qu L H， Luo X Y， Wang M N and Song Z J. 2022. Bi-directional weakly supervised knowledge distillation for whole slide image classification//Proceedings of the 36th Conference on Neural Information Processing Systems. New Orleans， USA： Curran Associates Inc.： 15368-15381
Shao Z C， Bian H， Chen Y， Wang Y F， Zhang J， Ji X Y and Zhang Y B. 2021. TransMIL： Transformer based correlated multiple instance learning for whole slide image classification//Proceedings of the 34th Conference on Neural Information Processing Systems. Sydney， Australia： Curran Associates Inc.： 2136-2147
Srinidhi C L， Kim S W， Chen F D and Martel A L. 2022. Self-supervised driven consistency training for annotation efficient histopathology image analysis. Medical Image Analysis， 75： #102256 ［DOI： 10.1016/j.media.2021.102256http://dx.doi.org/10.1016/j.media.2021.102256］
Su Z Y， Rezapour M， Sajjad U， Gurcan M N and Niazi M K K. 2023. Attention2Minority： a salient instance inference-based multiple instance learning for classifying small lesions in whole slide images. Computers in Biology and Medicine， 167： #107607 ［DOI： 10.1016/j.compbiomed.2023.107607http://dx.doi.org/10.1016/j.compbiomed.2023.107607］
Sung H， Ferlay J， Siegel R L， Laversanne M， Soerjomataram I， Jemal A and Bray F. 2021. Global cancer statistics 2020： GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA： A Cancer Journal for Clinicians， 71（3）： 209-249 ［DOI： 10.3322/caac.21660http://dx.doi.org/10.3322/caac.21660］
Tian X J， Liu G C， Gu S S， Ju Z J， Liu J G and Gu D D. 2018. Deep learning in medical image analysis and its challenges. Acta Automatica Sinica， 44（3）： 401-424
田娟秀， 刘国才， 谷珊珊， 鞠忠建， 刘劲光， 顾冬冬. 2018. 医学图像分析深度学习方法研究与挑战. 自动化学报， 44（3）： 401-424 ［DOI： 10.16383/j.aas.2018.c170153http://dx.doi.org/10.16383/j.aas.2018.c170153］
Vaswani A， Shazeer N， Parmar N， Uszkoreit J， Jones L， Gomez A N， Kaiser L and Polosukhin I. 2017. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach， USA： Curran Associates Inc.： 6000-6010
Wang D Y， Khosla A， Gargeya R， Irshad H and Beck A H. 2016. Deep learning for identifying metastatic breast cancer ［EB/OL］. ［2023-05-20］. https://arxiv.org/pdf/1606.05718.pdfhttps://arxiv.org/pdf/1606.05718.pdf
Wang X， Chen H， Gan C X， Lin H J， Dou Q， Tsougenis E， Huang Q T， Cai M Y and Heng P A. 2020. Weakly supervised deep learning for whole slide lung cancer image analysis. IEEE Transactions on Cybernetics， 50（9）： 3950-3962 ［DOI： 10.1109/TCYB.2019.2935141http://dx.doi.org/10.1109/TCYB.2019.2935141］
Wang X G， Yan Y L， Tang P， Bai X and Liu W Y. 2018. Revisiting multiple instance neural networks. Pattern Recognition， 74： 15-24 ［DOI： 10.1016/j.patcog.2017.08.026http://dx.doi.org/10.1016/j.patcog.2017.08.026］
Wang X Y， Yang S， Zhang J， Wang M H， Zhang J， Yang W， Huang J Z and Han X. 2022. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis， 81： #102559 ［DOI： 10.1016/j.media.2022.102559http://dx.doi.org/10.1016/j.media.2022.102559］
Wang Y W， Guo J， Yang Y， Kang Y， Xia Y L， Li Z H， Duan Y C and Wang K L. 2023. CWC-Transformer： a visual Transformer approach for compressed whole slide image classification. Neural Computing and Applications， 1-13 ［DOI： 10.1007/s00521-022-07857-3http://dx.doi.org/10.1007/s00521-022-07857-3］
Xie Z D， Lin Y T， Yao Z L， Zhang Z， Dai Q， Cao Y and Hu H. 2021. Self-supervised learning with swin Transformers ［EB/OL］. ［2023-05-20］. https://arxiv.org/pdf/2105.04553.pdfhttps://arxiv.org/pdf/2105.04553.pdf
Yan Y L， Wang X G， Guo X J， Fang J M， Liu W Y and Huang J Z. 2018. Deep multi-instance learning with dynamic pooling//Proceedings of the 10th Asian Conference on Machine Learning. Beijing， China： PMLR： 662-677
Zhao Y L， Ding W L， You Q H， Zhu F L， Zhu X J， Zheng K and Liu D D. 2023. Classification of whole slide images of breast histopathology based on spatial correlation characteristics. Journal of Image and Graphics， 28（4）： 1134-1145
赵樱莉， 丁维龙， 游庆华， 朱峰龙， 朱筱婕， 郑魁， 刘丹丹. 2023. 融合空间相关性特征的乳腺组织病理全切片分类. 中国图象图形学报， 28（4）： 1134-1145 ［DOI： 10.11834/jig.211133http://dx.doi.org/10.11834/jig.211133］
Zhang H R， Meng Y D， Zhao Y T， Qiao Y H， Yang X Y， Coupland S E and Zheng Y L. 2022. DTFD-MIL： double-tier feature distillation multiple instance learning for histopathology whole slide image classification//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans， USA： IEEE： 18802-18812 ［DOI： 10.1109/CVPR52688.2022.01824http://dx.doi.org/10.1109/CVPR52688.2022.01824］
相关作者
相关机构
京公网安备11010802024621