浏览全部资源
扫码关注微信
收稿：2022-05-16，
修回：2022-7-5，
录用：2022-7-12，
纸质出版：2023-01-16
移动端阅览
目的
2
计算机辅助诊断是临床诊断中一种重要的辅助手段。然而在多机型超声影像的应用现状中，单一深度卷积神经网络面临难以从不同数据源中提取样本特征的问题，导致模型在区分多源数据方面性能欠佳。为提升单一深度模型在多源数据的泛化能力，本文提出一种无监督域自适应网络。
方法
2
将深度对抗域适应方法应用于多源甲状腺超声影像分类任务，通过生成对抗思想提取源域图像与目标域图像的域不变特征，提出一种多级对抗域自适应网络(multi-level adversarial domain adaptation network，MADAN)。将元优化(meta-optimized)策略引入对抗域适应的学习中，将域对齐目标和样本分类目标以协调的方式联合优化，提升了模型对无标记目标域数据的分类性能。
结果
2
在包含4种域的甲状腺超声影像数据集上实验，与7种经典域自适应方法比较。实验结果表明，MADAN在全部迁移任务中取得90.141%的目标域样本平均分类准确率，优于残差分类网络和多种经典域自适应分类网络。融合元优化训练策略后的MADAN在目标域的测试平均准确率提升约1.67%。
结论
2
本文提出的元优化多级对抗域适应网络一方面通过多级对抗学习进行图像域不变特征的提取，另一方面使用元优化方式改进模型训练过程的优化策略，将带有人工标记的源域信息有效迁移至目标域，提升了单一模型对于不同域数据的泛化性能。
Objective
2
Artificial intelligence based (AI-based) medical clinical diagnosis technique has been developing in recent years. It can alleviate the problems of medical image analysis in China via deep networks modeling and medical image data analysis
such as the shortage of expertises
the imbalance of urban and rural medical resources allocation
and the the imaging accuracy issues. However
clinical-aided are often linked to multi-model data with different characteristics distribution in different hospitals. Therefore
improving the generalization and stability of the cross-model medical diagnosis-consistent model is required for quick response intensively. In order to alleviate the domain shift existing in the ultrasound imaging field
the unsupervised domain adaptation method can be as the one of the most concerning methods at present. It can avoid the manual labeling of ultrasound image data of various models
a single model can be learnt to adapt to the target domain sample set with data deviation through the labeled source domain sample set
which improves the generalizability of convolutional neural network (CNN) to a certain extent. However
current unsupervised domain adaptation research has some challenging constraints
such as poor feature extraction and inconsistent optimization of domain fusion and sample classification. In view of the limitations in related to domain adaptation network
we develop an intergrated domain adaptation network
which focuses on the under-expressed feature of nodular region features in thyroid ultrasound images. This research is aimed to enhance the fusion of source domain features space and target domain features space.
Method
2
In this study
a new domain adaptation network is constructed called based on domain-adversarial training of neural networks (DANN)
called multi-level adversarial domain adaptation netwo
rk (MADAN). In the training process
we first build a three-layer generator and discriminator structure according to the transition from general to special features
which can obtain more semantic information in the image. To ensure the coordination of the two optimization processes
the sample classification task and the domain fusion task are both implemented via a meta-optimization strategy. Furthermore
to enhance the global geometric features of diseased tissue regions
we embed a self-attention module in the adversarial structure. Our dataset is composed of 6 849 images-selected from ultrasound images in 4 different models
namely
P
(Philips1)
T
(Toshiba)
F
(GE)
and
U
(Philips2). Our domain adaptation network is carried out in the PyTorch toolbox. The input image is resized to 224 × 224 pixels for training
the momentum parameter is set to 0.9
and the learning rate is set to 0.001. The stochastic gradient descent (SGD)-related learning process is accelerated using an NVIDIA GTX TITAN RTX device
which takes approximately 10 hours in 50 000 iterations. It is possible to obtain more effective samples in the context to benign and malignant labels of the source domain
the target domain during the training process
and the labels of the target domain data are not required at all. The training process performs the supervised classification task of the source domain dataset and the domain alignment task. In the testing process
the feature extractor and classifier fitted are used to verify the classification accuracy of the target domain data. In addition
we use t-distributed stochastic neighbor embedding (t-SNE) to visualize the learned data features
which demonstrates the effectiveness of our network further on the feature-fused of different datasets.
Result
2
We compared our network with 8 domain adaptation networks
including domain adaptation tasks based on 9 multi-directions on 4 domains:
P
→
T
P
→
U
P
→
F
T
→
P
T
→
U
T→F
U
→
P
U
→
T
and
U
→
F
. The evaluation metrics are focused on the accuracy of the target domain data and the t-SNE visualization results
and the ablation experiments is conducted to demonstrate the performance and GPU memory cost of the proposed method. Experimental results show that the MADAN is optimized with an average classification accuracy of 90.141% on the multi-model thyroid ultrasound dataset through 9 transfer tasks. After introducing meta-optimization
the best classification accuracy is achieved in
P
→
U
P
→
F
T
→
P
T
→
F
U
→
P
and
U
→
F
tasks respectively
and the average accuracy of all tasks is improved about 1.67%. The results of the ablation experiments illustrates that the domain adaptation
multi-adversarial structure
and self-attention modules are improved 16.238%
20.284%
and 18.622%
respectively. The t-SNE visualization images are illustrated that the feature space of the data samples has been preliminarily fused after the multi-level adversarial domain adaptation
and the sample points of the same category are basically aggregated. The multi-adversarial domain adaptation method can achieve fusion results better via the improved meta-optimization strategy. Additionally
the comparative visualization analysis of the heat map verifies the effectiveness of the self-attention module for global geometric feature extraction of the lesion tissue.
Conclusion
2
Our research is focused on meta-optimized multi-Adversarial domain adaptation network
including a multi-level generative adversarial structure and a meta-optimization strategy. The experimental results show that the proposed method is performed well in the transfer task of multi-model medical thyroid ultrasound datasets and has better classification results in a completed unsupervised setting.
Andrychowicz M, Denil M, Colmenarejo S G, Hoffman M W, Pfau D, Schaul T, Shillingford B and de Freitas N. 2016. Learning to learn by gradient descent by gradient descent//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain: Curran Associates Inc. : 3988-3996
Chi J N, Yu X S and Zhang Y F. 2018. Thyroid nodule malignantrisk detection in ultrasound image by fusing deep and texture features. Journal of Image and Graphics, 23(10): 1582-1593
迟剑宁, 于晓升, 张艺菲. 2018. 融合深度网络和浅层纹理特征的甲状腺结节癌变超声图像诊断. 中国图象图形学报, 23(10): 1582-1593 [DOI: 10.11834/jig.180232]
Dong N Q, Kampffmeyer M, Liang X D, Wang Z Y, Dai W and Xing E. 2018. Unsupervised domain adaptation for automatic estimation of cardiothoracic ratio//Proceedings of the 21st International Conference on Medical Image Computing and Computer Assisted Intervention. Granada, Spain: Springer: 544-552 [ DOI: 10.1007/978-3-030-00934-2_61 http://dx.doi.org/10.1007/978-3-030-00934-2_61 ]
Finn C, Abbeel P and Levine S. 2017. Model-agnostic meta-learning for fast adaptation of deep networks//Proceedings of the 34th International Conference on Machine Learning. Sydney, Australia: PMLR: 1126-1135
Ganin Y and Lempitsky V. 2015. Unsupervised domain adaptation by backpropagation//Proceedings of the 32nd International Conference on Machine Learning. Lille, France: JMLR: 1180-1189
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M and Lempitsky V S. 2016. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59): 1-35 [DOI: 10.1007/978-3-319-58347-1_10]
Gong R L, Shi J, Zhou W J and Wang C. 2022. Two-stage deep transfer learning for human breast ultrasound computer-aided diagnosis. Journal of Image and Graphics, 27(3): 898-910
贡荣麟, 施俊, 周玮珺, 汪程. 2022. 面向乳腺超声计算机辅助诊断的两阶段深度迁移学习. 中国图象图形学报, 27(3): 898-910 [DOI: 10.11834/jig.210674]
Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A C and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press: 2672-2680
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778 [ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Li D and Hospedales T M. 2020. Online meta-learning for multi-source and semi-supervised domain adaptation//Proceedings of the 16th European Conference on Computer Vision. Glasgow, UK: Springer: 382-403 [ DOI: 10.1007/978-3-030-58517-4_23 http://dx.doi.org/10.1007/978-3-030-58517-4_23 ]
Long M S, Cao Y, Wang J M and Jordan M I. 2015. Learning transferable features with deep adaptation networks//Proceedings of the 32nd International Conference on Machine Learning. Lille, France: JMLP: 97-105 [ DOI:10.48550/arXiv.1502.02791 http://dx.doi.org/10.48550/arXiv.1502.02791 ]
Long M S, Cao Z J, Wang J M and Jordan M I. 2018. Conditional Adversarial Domain Adaptation//Proceedings of the 31st International Conference on Neural Information Processing Systems. Montreal, Canada. : Curran Associates Inc. : 1647-1657. [ DOI: 10.48550/arXiv.1705.10667 http://dx.doi.org/10.48550/arXiv.1705.10667 ]
Ma J L, Wu F, Jiang T A, Zhu J and Kong D X. 2017. Cascade convolutional neural networks for automatic detection of thyroid nodules in ultrasound images. Medical Physics, 44(5): 1678-1691 [DOI: 10.1002/mp.12134]
Pacheco C and Vidal R. 2019. An unsupervised domain adaptation approach to classification of stem cell-derived cardiomyocytes//Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention. Shenzhen, China: Springer: 806-814 [ DOI: 10.1007/978-3-030-32239-7_89 http://dx.doi.org/10.1007/978-3-030-32239-7_89 ]
Pan S J and Yang Q. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10): 1345-1359 [DOI: 10.1109/TKDE.2009.191]
Pei Z Y, Cao Z J, Long M S and Wang J M. 2018. Multi-adversarial domain adaptatio n//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press: 3934-3941 [ DOI: 10.1609/aaai.v32i1.11767 http://dx.doi.org/10.1609/aaai.v32i1.11767 ]
Saenko K, Kulis B, Fritz M and Darrell T. 2010. Adapting visual category models to new domains//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece: Springer: 213-226 [ DOI: 10.1007/978-3-642-15561-1_16 http://dx.doi.org/10.1007/978-3-642-15561-1_16 ]
Song R N, Zhu C, Zhang L, Zhang T, Luo Y H, Liu J and Yang J. 2022. Dual-branch network via pseudo-label training for thyroid nodule detection in ultrasound image. Applied Intelligence, 52(10): 11738-11754 [DOI: 10.1007/s10489-021-02967-2]
Sun B C and Saenko K. 2016. Deep CORAL: correlation alignment for deep domain adaptation//Proceedings of 2016 European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 443-450 [ DOI: 10.1007/978-3-319-49409-8_35 http://dx.doi.org/10.1007/978-3-319-49409-8_35 ]
Sun C, Zhang Y K, Chang Q, Liu T J, Zhang S H, Wang X, Guo Q Q, Yao J P, Sun W D and Niu L J. 2020. Evaluation of a deep learning-based computer-aided diagnosis system for distinguishing benign from malignant thyroid nodules in ultrasound images. Medical Physics, 47(9): 3952-3960 [DOI: 10.1002/mp.14301]
Tessler F N, Middleton W D, Grant E G, Hoang J K, Berland L L, Teefey S A, Cronan J J, Beland M D, Desser T S, Frates M C, Hammers L W, Hamper U M, Langer J E, Reading C C, Scoutt L M and Stavros A T. 2017. ACR thyroid imaging, reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee. Journal of the American College of Radiology, 14(5): 587-595 [DOI: 10.1016/j.jacr.2017.01.046]
Tzeng E, Hoffman J, Saenko K and Darrell T. 2017. Adversarial discriminative domain adaptation//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2962-2971 [ DOI: 10.1109/CVPR.2017.316 http://dx.doi.org/10.1109/CVPR.2017.316 ]
Tzeng E, Hoffman J, Zhang N, Saenko E and Darrell T. 2014. Deep domain confusion: maximizing for domain invariance [EB/OL ] . [2022-05-08 ] . https://arxiv.org/pdf/1412.3474.pdf https://arxiv.org/pdf/1412.3474.pdf
Vinyals O, Blundell C, Lillicrap T, Kavukcuoglu K and Wierstra D. 2016. Matching networks for one shot learning//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain: Curran Associates Inc. : 3637-3645
Wang S J, Yu L Q, Li K, Yang X, Fu C W and Heng P A. 2019. Boundary and entropy-driven adversarial learning for fundus image segmentation//Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention. Shenzhen, China: Springer: 102-110 [ DOI: 10.1007/978-3-030-32239-7_12 http://dx.doi.org/10.1007/978-3-030-32239-7_12 ]
Wang X L, Girshick R, Gupta A and He K M. 2018. Non-local neural networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7794-7803 [ DOI: 10.1109/CVPR.2018.00813 http://dx.doi.org/10.1109/CVPR.2018.00813 ]
Xu M H, Zhang J, Ni B B, Li T, Wang C J, Tian Q and Zhang W J. 2020. Adversarial domain adaptation with domain mixup. Proceedings of 2020 AAAI Conference on Artificial Intelligence, 34(4): 6502-6509 [DOI: 10.1609/aaai.v34i04.6123]
Ying X, Zhang Y L, Wei X, Yu M, Zhu J L, Gao J, Liu Z Q, Li X W and Yu R G. 2020. MSDAN: multi-scale self-attention unsupervised domain adaptation network for thyroid ultrasound images//Proceedings of 2020 IEEE International Conference on Bioinformatics and Biomedicine. Seoul, Korea (South): IEEE: 871-876 [ DOI: 10.1109/BIBM49941.2020.9313202 http://dx.doi.org/10.1109/BIBM49941.2020.9313202 ]
Yu M, Han M, Li X W, Wei X, Jiang H, Chen H L and Yu R G. 2022. Adaptive soft erasure with edge self-attention for weakly supervised semantic segmentation: thyroid ultrasound image case study. Computers in Biology and Medicine, 144: #105347 [DOI: 10.1016/j.compbiomed.2022.105347]
Yu X, Wang H J and Ma L Y. 2020. Detection of thyroid nodules with ultrasound images based on deep learning. Current Medical Imaging, 16(2): 174-180 [DOI: 10.2174/1573405615666191023104751]
Zhang Y L, Hu Y, Higashita R and Liu J. 2022. A review of generative adversarial networks and the application in medical image. Journal of Image and Graphics, 27(3): 687-703
张颖麟, 胡衍, 东田理沙, 刘江. 2022. 生成对抗式网络及其医学影像应用研究综述. 中国图象图形学报, 27(3): 687-703 [DOI: 10.11834/jig.210247]
Zhao J, Zhou X S, Shi G H, Xiao N, Song K, Zhao J J, Hao R and Li K Q. 2022. Semantic consistency generative adversarial network for cross-modality domain adaptation in ultrasound thyroid nodule classification. Applied Intelligence, 52(9): 10369-10383 [DOI: 10.1007/s10489-021-03025-7]
Zhu Y C, Zhuang F Z, Wang J D, Chen J W, Shi Z P, Wu W J and He Q. 2019. Multi-representation adaptation network for cross-domain image classification. Neural Networks, 119: 214-221 [DOI: 10.1016/j.neunet.2019.07.010]
相关作者
相关机构
京公网安备11010802024621
