浏览全部资源
扫码关注微信
Published： 16 September 2023 ，
移动端阅览
林凡超， 谢洪涛， 刘传彬， 张勇东. 2023. 外观和运动模式感知的有丝分裂细胞检测. 中国图象图形学报， 28(09):2733-2748
Lin Fanchao， Xie Hongtao， Liu Chuanbin， Zhang Yongdong. 2023. Mitosis detection by appearance and motion pattern perception. Journal of Image and Graphics， 28(09):2733-2748
目的
2
在癌症筛查和药物研发等医学研究和诊疗过程中，显微图像中的有丝分裂细胞检测可以提供重要的生物学判据。然而，不同培养条件下图像分布差异明显，且细胞密度逐渐增大导致场景变得复杂，常规预处理方法难以进行有效的区域筛选；不同阶段细胞外观相似、运动过程模糊，现有方法缺乏对区域特征编码的显式监督，容易因为语义区分能力不足导致错误预测。为此，本文提出基于外观和运动模式感知的检测框架，通过两阶段预处理和对细胞状态模式的判别性学习，实现复杂场景下的精准预测。
方法
2
本文方法采用3阶段检测框架：在预处理阶段结合区域分割网络和先验优化算法来充分精简候选区域；在预训练阶段构造基于图像分类和重构的两种辅助任务，为候选区域的外观和运动编码提供直接监督，使编码网络具备对不同细胞状态的语义感知能力；在全模型训练和预测阶段，以预处理得到的候选区域序列作为输入，用预训练的编码网络提取候选区域特征，最终通过时序网络融合序列上下文信息得到细胞检测结果。
结果
2
在C2C12-16数据集上的实验结果表明，本文方法的平均性能达到：验证集精准率85.3%，召回率89.3%，F得分87.2%；测试集精准率86.4%，召回率86.1%，F得分86.2%，时序检测误差0.221±0.536帧，空间检测误差3.321±2.461像素，在检测精度和稳定性上都超过了现有方法。
结论
2
本文提出了新的复杂场景下有丝分裂细胞检测框架。所采用的预处理策略可以有效精简候选区域，显著提高检测效率；针对编码网络的辅助任务预训练充分提升了模型对候选区域外观和运动特征的学习能力，最终能够克服电镜图像中复杂场景和细胞模式的干扰，准确且稳定地对有丝分裂细胞进行时空检测。
Objective
2
In the processes of medical research and diagnosis， such as cancer screening and drug development， mitosis detection under phase-contrast microscopy image provides a very important biological criterion. Manual counting of mitotic cells takes a lot of time and labor. Thus， automatic mitosis detection is more efficient and economic than the manual process. On the one hand， the distributions of mitosis images are significantly different under various culture conditions. Moreover， the increment of cell density makes screening out the cell regions difficult for conventional preprocessing methods. On the other hand， the cells at different stages have similar appearances and blurred motion processes. They also require the model to have a strong ability to discriminate cell types and states. Recent deep-learning-based works use three-dimensional convolutions or temporal networks to obtain context information from the sequence images. However， an explicit supervision process for learning cell states is lacking， making effective pattern information of target regions difficult to achieve. As a result， these methods are not fully capable of distinguishing different cells and background areas from feature encoding， and their performance and generalization ability are limited. Therefore， this study explores a detection framework based on cell appearance and motion pattern perception to solve the above problems. An accurate prediction under complex scenes is also achieved through effective preprocessing and discriminative learning of cell patterns.
Method
2
The proposed method consists of three stages. The first stage aims to extract regions of interest as candidates. This stage serves as the preprocessing for finding the notable areas and facilitating the later detection. The original electron microscope image is divided into local slices. An instance segmentation network is also trained to segment roughly all the candidate regions that may contain mitosis. Then， a candidate region refinement algorithm is designed based on a concise spatiotemporal hypothesis to refine the candidates and reduce the redundant results. In the second stage， two encoding networks are pretrained to maintain the feature encoding of both appearance and motion information by building proxy learning processes. In particular， an image classification task is conducted for the appearance encoding network training， which learns to predict the cell categories from the spatial context of a single patch. Moreover， an image reconstruction task is conducted for the motion encoding network training， which considers patches from adjacent frames and learns the information of interframe changes by recovering the raw patches. These two processes complement each other to help model the cell states from different aspects. Finally， in the third stage， the whole spatiotemporal model is trained end-to-end by classifying the candidate patch sequences. The spatial modules are initialized with the pretrained parameters of encoding networks in the second stage， thereby allowing them to be aware of the cell patterns at the beginning of the training. Given the appropriate spatial context， the temporal modules are optimized to combine the interframe information and make the final prediction. The overall model provides a confidence score for each patch. The position with the highest score is regarded as a mitosis point.
Result
2
We conduct experiments on the public C2C12-16 benchmark. The experimental results demonstrate the superior detection ability of the proposed method. On the C2C12-16 validation set， the mean precision reaches 85.3%， the mean recall reaches 89.3%， and the mean F-score is 87.2%. On the C2C12-16 test set， the mean precision reaches 86.4%， the mean recall reaches 86.1%， and the F-score is 86.2%. The proposed method demonstrates high performance and can generate stable predictions under various conditions. The mean temporal bias of the proposed method in all groups is only 0.221 ± 0.536 frames， and the mean spatial bias is 3.321 ± 2.461 pixels， both of which are much lower than those obtained by the counterpart method.
Conclusion
2
This study explores a new framework to tackle the hard cases under complex scenes in mitosis detection. The preprocessing strategy effectively extracts candidate regions and substantially improves detection efficiency. The pre-training of the feature encoding network based on proxy tasks fully enhances the model’s ability to learn the appearance and motion patterns of the candidate regions. With the preprocessing and pretraining designs， our framework can distinguish the discrepancy of visual patterns between mitosis cells， common cells， and background noises， overcome the interference of complex scenes and cell patterns in the microscope image， and achieve both accurate and stable mitosis detection from spatiotemporal dimensions.
相衬显微图像有丝分裂细胞检测多阶段检测时空特征编码辅助训练
phase contrast microscopy imagemitosis detectionmultistage detectionspatiotemporal feature encodingproxy training
Cao J L， Li Y L， Sun H Q， Xie J， Huang K Q and Pang Y W. 2022. A survey on deep learning based visual object detection. Journal of Image and Graphics， 27（6）： 1697-1722
曹家乐， 李亚利， 孙汉卿， 谢今， 黄凯奇， 庞彦伟. 2022. 基于深度学习的视觉目标检测技术综述. 中国图象图形学报， 27（6）： 1697-1722 ［DOI： 10.11834/jig.220069http://dx.doi.org/10.11834/jig.220069］
Chen L C， Zhu Y K， Papandreou G， Schroff F and Adam H. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 833-851 ［DOI： 10.1007/978-3-030-01234-2_49http://dx.doi.org/10.1007/978-3-030-01234-2_49］
Debeir O， van Ham P， Kiss R and Decaestecker C. 2005. Tracking of migrating cells under phase-contrast video microscopy with combined mean-shift processes. IEEE Transactions on Medical Imaging， 24（6）： 697-711 ［DOI： 10.1109/TMI.2005.846851http://dx.doi.org/10.1109/TMI.2005.846851］
Gallardo G M， Yang F X， Ianzini F， Mackey M and Sonka M. 2004. Mitotic cell recognition with hidden Markov models//Proceedings of SPIE 5367， Medical Imaging 2004： Visualization， Image-Guided Procedures， and Display. San Diego， USA： SPIE： 661-668 ［DOI： 10.1117/12.535778http://dx.doi.org/10.1117/12.535778］
Hochreiter S and Schmidhuber J. 1997. Long short-term memory. Neural Computation， 9（8）： 1735-1780 ［DOI： 10.1162/neco.1997.9.8.1735http://dx.doi.org/10.1162/neco.1997.9.8.1735］
Huh S， Ker DFE， Bise R， Chen M and Kanade T. 2011. Automated mitosis detection of stem cell populations in phase-contrast microscopy images. IEEE Transactions on Medical Imaging， 30（3）： 586-596 ［DOI： 10.1109/TMI.2010.2089384http://dx.doi.org/10.1109/TMI.2010.2089384］
Jiang X， Yuan Y X， Wang Y P， Xiao Z X， Zhu M L， Chen Z H， Liu T M and Shen D G. 2022. Review and prospect of medical imaging artificial intelligence in China in the past 20 years. Journal of Image and Graphics， 27（3）： 655-671
蒋希， 袁奕萱， 王雅萍， 肖振祥， 朱美芦， 陈泽华， 陈泽华， 刘天明， 沈定刚. 2022. 中国医学影像人工智能20年回顾和展望. 中国图象图形学报， 27（3）： 655-671 ［DOI： 10.11834/jig.211162http://dx.doi.org/10.11834/jig.211162］
Li K and Kanade T. 2009. Nonnegative mixed-norm preconditioning for microscopy image segmentation//Proceedings of the 21st International Conference on Information Processing in Medical Imaging. Williamsburg， USA： Springer： 362-373 ［DOI： 10.1007/978-3-642-02498-6_30http://dx.doi.org/10.1007/978-3-642-02498-6_30］
Li K， Miller E D， Chen M， Kanade T， Weiss L E and Campbell P G. 2008. Computer vision tracking of stemness//Proceedings of the 5th IEEE International Symposium on Biomedical Imaging： from Nano to Macro. Paris， France： IEEE： 847-850 ［DOI： 10.1109/ISBI.2008.4541129http://dx.doi.org/10.1109/ISBI.2008.4541129］
Li Y H， Wu C Y， Fan H Q， Mangalam K， Xiong B， Malik J and Feichtenhofer C. 2022. MViTv2： improved multiscale vision transformers for classification and detection//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans， USA： IEEE： 4794-4804 ［DOI： 10.1109/cvpr52688.2022.00476http://dx.doi.org/10.1109/cvpr52688.2022.00476］
Liang L C， Zhou X B， Li F H， Wong S T C， Huckins J and King R W. 2007. Mitosis cell identification with conditional random fields//Proceedings of 2007 IEEE/NIH Life Science Systems and Applications Workshop. Bethesda， USA： IEEE： 2007： 9-12 ［DOI： 10.1109/lssa.2007.4400872http://dx.doi.org/10.1109/lssa.2007.4400872］
Liu A A， Hao T， Gao Z， Su Y T and Yang Z X. 2013. Nonnegative mixed-norm convex optimization for mitotic cell detection in phase contrast microscopy. Computational and Mathematical Methods in Medicine， 2013： #176272 ［DOI： 10.1155/2013/176272http://dx.doi.org/10.1155/2013/176272］
Liu A A， Li K and Kanade T. 2010a. Spatiotemporal mitosis event detection in time-lapse phase contrast microscopy image sequences//Proceedings of 2010 IEEE International Conference on Multimedia and Expo. Singapore， Singapore： IEEE： 161-166 ［DOI： 10.1109/ICME.2010.5583299http://dx.doi.org/10.1109/ICME.2010.5583299］
Liu A A， Li K and Kanade T. 2010b. Mitosis sequence detection using hidden conditional random fields//Proceedings of 2010 IEEE International Symposium on Biomedical Imaging： from Nano to Macro. Rotterdam， the Netherlands： IEEE： 580-583 ［DOI： 10.1109/ISBI.2010.5490279http://dx.doi.org/10.1109/ISBI.2010.5490279］
Liu A A， Li K and Kanade T. 2012. A semi-Markov model for mitosis segmentation in time-lapse phase contrast microscopy image sequences of stem cell populations. IEEE Transactions on Medical Imaging， 31（2）： 359-369 ［DOI： 10.1109/TMI.2011.2169495http://dx.doi.org/10.1109/TMI.2011.2169495］
Liu A A， Lu Y， Chen M and Su Y T. 2017a. Mitosis detection in phase contrast microscopy image sequences of stem cell populations： a critical review. IEEE Transactions on Big Data， 3（4）： 443-457 ［DOI： 10.1109/TBDATA.2017.2721438http://dx.doi.org/10.1109/TBDATA.2017.2721438］
Liu M， He Y， Wei Y L and Xiang P. 2017b. Plant cell tracking using Kalman filter based local graph matching. Image and Vision Computing， 60： 154-161 ［DOI： 10.1016/j.imavis.2016.08.005http://dx.doi.org/10.1016/j.imavis.2016.08.005］
Lu Y， Liu A A， Chen M， Nie W Z and Su Y T. 2020. Sequential saliency guided deep neural network for joint mitosis identification and localization in time-lapse phase contrast microscopy images. IEEE Journal of Biomedical and Health Informatics， 24（5）： 1367-1378 ［DOI： 10.1109/JBHI.2019.2943228http://dx.doi.org/10.1109/JBHI.2019.2943228］
Mao Y X and Yin Z Z. 2016. A hierarchical convolutional neural network for mitosis detection in phase-contrast microscopy images//Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention. Athens， Greece： Springer： 685-692 ［DOI： 10.1007/978-3-319-46723-8_79http://dx.doi.org/10.1007/978-3-319-46723-8_79］
Marcuzzo M， Guichard T， Quelhas P， Mendonça A M and Campilho A. 2009. Cell division detection on the arabidopsis thaliana root//Proceedings of the 4th Iberian Conference on Pattern Recognition and Image Analysis. Póvoa de Varzim， Portugal： Springer： 168-175 ［DOI： 10.1007/978-3-642-02172-5_23http://dx.doi.org/10.1007/978-3-642-02172-5_23］
Matsoukas C， Haslum J F， Söderberg M and Smith K. 2021. Is it time to replace CNNs with transformers for medical images？ ［EB/OL］. ［2021-08-20］. https://arxiv.org/pdf/2108.09038.pdfhttps://arxiv.org/pdf/2108.09038.pdf
Milletari F， Navab N and Ahmadi S A. 2016. V-Net： fully convolutional neural networks for volumetric medical-image segmentation//Proceedings of the 4th International Conference on 3D Vision. Stanford， USA： IEEE： 565-571 ［DOI： 10.1109/3DV.2016.79http://dx.doi.org/10.1109/3DV.2016.79］
Nie W Z， Li W H， Liu A A， Hao T and Su Y T. 2016. 3D convolutional networks-based mitotic event detection in time-lapse phase contrast microscopy image sequences of stem cell populations//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Las Vegas， USA： IEEE： 1359-1366 ［DOI： 10.1109/cvprw.2016.171http://dx.doi.org/10.1109/cvprw.2016.171］
Nishimura K and Bise R. 2020. Spatial-temporal mitosis detection in phase-contrast microscopy via likelihood map estimation by 3DCNN//Proceedings of the 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Montreal， Canada： IEEE： 1811-1815 ［DOI： 10.1109/EMBC44109.2020.9175676http://dx.doi.org/10.1109/EMBC44109.2020.9175676］
Quelhas P， Mendonça A M and Campilho A. 2010. Optical flow based arabidopsis thaliana root meristem cell division detection//Proceedings of the 7th International Conference on Image Analysis and Recognition. Póvoa de Varzim， Portugal： Springer： 217-226 ［DOI： 10.1007/978-3-642-13775-4_22http://dx.doi.org/10.1007/978-3-642-13775-4_22］
Raghu M， Unterthiner T， Kornblith S， Zhang C Y and Dosovitskiy A. 2021. Do vision transformers see like convolutional neural networks?//Proceedings of the 35th Conference on Neural Information Processing Systems. ［s.l.］： NeurIPS： 12116-12128
Su Y T， Lu Y， Chen M and Liu A A. 2017. Spatiotemporal joint mitosis detection using CNN-LSTM network in time-lapse phase contrast microscopy images. IEEE Access， 5： 18033-18041 ［DOI： 10.1109/ACCESS.2017.2745544http://dx.doi.org/10.1109/ACCESS.2017.2745544］
Su Y T， Lu Y， Chen M and Liu A A. 2022. Deep reinforcement learning-based progressive sequence saliency discovery network for mitosis detection in time-lapse phase-contrast microscopy images. IEEE/ACM Transactions on Computational Biology and Bioinformatics， 19（2）： 854-865 ［DOI： 10.1109/TCBB.2020.3019042http://dx.doi.org/10.1109/TCBB.2020.3019042］
Su Y T， Lu Y， Liu J， Chen M and Liu A A. 2021. Spatio-temporal mitosis detection in time-lapse phase-contrast microscopy image sequences： a benchmark. IEEE Transactions on Medical Imaging， 40（5）： 1319-1328 ［DOI： 10.1109/TMI.2021.3052854http://dx.doi.org/10.1109/TMI.2021.3052854］
Yang F X， Mackey M A， Ianzini F， Gallardo G and Sonka M. 2005. Cell segmentation， tracking， and mitosis detection using temporal context//Proceedings of the 8th International Conference on Medical Image Computing and Computer-Assisted Intervention. Palm Springs， USA： Springer： 302-309 ［DOI： 10.1007/11566465_38http://dx.doi.org/10.1007/11566465_38］
相关文章
相关作者
相关机构
京公网安备11010802024621