浏览全部资源
扫码关注微信
Received：12 April 2019，
Revised：2019-9-16，
Accepted：23 September 2019，
Published：16 April 2020
移动端阅览
目的
2
针对图像合成配准算法中鲁棒性差及合成图像特征信息不足导致配准精度不高的问题，提出了基于残差密集相对平均条件生成对抗网络（residual dense-relativistic average conditional generative adversarial network，RD-RaCGAN）的多模态脑部图像配准方法。
方法
2
相对平均生成对抗网络中的相对平均鉴别器能够增强模型稳定性，条件生成对抗网络加入条件变量能提高生成数据质量，结合两种网络特点，利用残差密集块充分提取深层网络特征的能力，构建RD-RaCGAN合成模型。然后，待配准的参考CT（computed tomography）和浮动MR（magnetic resonance）图像通过已训练好的RD-RaCGAN合成模型双向合成对应的参考MR和浮动CT图像。采用区域自适应配准算法，从参考CT和浮动CT图像中选取骨骼信息的关键点，从浮动MR和参考MR图像中选取软组织信息的关键点，通过提取的关键点指导形变场的估计。从浮动CT图像到参考CT图像估计一个形变场。类似地，从浮动MR图像到参考MR图像估计一个形变场。另外，采用分层对称的思想进一步优化两个形变场，当两个形变场之间的差异达到最小时，将两个形变场融合得到最终的形变场，并将形变场作用于浮动图像完成配准。
结果
2
实验结果表明，与其他6种图像合成方法相比，本文模型合成的目标图像在视觉效果和客观评价指标上均优于其他方法。对比Powell优化的MI（mutual information）法、ANTs-SyN（advanced normalization toolbox-symmetric normalization）、D.Demons（diffeomorphic demons）、Cue-Aware Net（cue-aware deep regression network）和I-SI（intensity and spatial information）的图像配准方法，归一化互信息分别提高了43.71%、12.87%、10.59%、0.47%、5.59%，均方根误差均值分别下降了39.80%、38.67%、15.68%、4.38%、2.61%。
结论
2
本文提出的多模态脑部图像配准方法具有很强的鲁棒性，能够稳定、准确地完成图像配准任务。
Objective
2
Multimodal medical image registration is a key step in medical image analysis and processing as it complements the information from different modality images and provides doctors with a variety of information about diseased tissues or organs. This method enables doctors to make accurate diagnosis and treatment plans. Image registration based on image synthesis is the main method for achieving high-precision registration. A high-quality composite image indicates good registration effect. However
current image-based registration algorithm has poor robustness in synthetic models and provides an insufficient representation of synthetic image feature information
resulting in low registration accuracy. In recent years
owing to the success of deep learning in many fields
medical image registration based on deep learning has become a focus of research. The synthetic model is trained according to the modal type of the image to be registered
and the synthetic model bidirectional synthetic image is used to guide the subsequent registration. Anatomical information is employed to guide the registration and improve the accuracy of multimodal image registration. Therefore
a multimodal brain image registration method based on residual dense relative average conditional generative adversarial network (RD-RaCGAN) is proposed in this study.
Method
2
First
the RD-RaCGAN image synthesis model is constructed by combining the advantages of the relative average discriminator in the relativistic average generative adversarial network
which can enhance the model stability
and the advantages of the conditional generative adversarial network
which can improve the quality of the generated data
and also the ability of residual dense blocks to fully extract the characteristics of the deep network. Residual dense blocks are utilized as core components for building a generator. The purpose of this generator is to capture the law of sample distribution and generate a target image with specific significance
that is
to input a floating magnetic resonance (MR) or reference computed tomography (CT) image and generate the corresponding synthetic CT or synthetic MR image. The convolution neural network is used as a relative average discriminator
which correctly distinguishes an image generated by the generator from the real image. The generator and relative average discriminator perform confrontational training. First
the generator is fixed to train the relative average discriminator.Then
the relative average discriminator is fixed to train the generator
and the loop training is subsequently continued. During training
the least square function optimization generator and relative average discriminator
which are more stable and less saturated than the cross entropy function are selected. The ability of the generator and the relative average discriminator is enhanced
and the image generated by the generator can be falsified. At this point
the synthetic model training is completed. Subsequently
the CT image and MR image to be registered are bidirectionally synthesized into the corresponding reference MR image and floating CT image through the RD-RaCGAN synthesis model that has been trained. Four images obtained by bidirectional synthesis are registered by a region-adaptive registration algorithm. Specifically
the key points of the bone information are selected from the reference CT image and the floating CT image.The key points of the soft tissue information are selected from the floating MR image and the reference MR image
and the estimation of the deformation field is guided by the extracted key points. In other words
one deformation field is estimated from the floating CT image to the reference CT image
and a deformation field is estimated from the floating MR image to the reference MR image. At the same time
the idea of hierarchical symmetry is adopted to further guide the registration. The key points in the image are gradually increased when the reference and floating images are close to each other.Moreover
anatomical information is used to optimize the two deformation fields continuously until the difference between the two deformation fields reaches a minimum. The two deformation fields are fused to form the deformation field between the reference CT image and floating MR image. Finally
the deformation field is applied to the floating image to complete registration. Given that the synthesis of a target image from two images to be registered through the synthesis model requires time
the algorithm efficiency in this study is slightly lower than that of D.Demons(diffeomorphic demons) and ANTs-SyN(advanced normalization toolbox-symmetric normalization).
Result
2
Given that the quality of the synthesized image directly affects registration accuracy
three sets of contrast experiments are designed to verify the effect of the algorithm in this study. Different algorithms are by MR synthesis CT
and different algorithms are compared by CT synthesis MR
and comparison of the effect of different registration algorithms. The experimental results show that the target image synthesized by the synthesis model in this study is superior to those obtained by the other methods in terms of visual effect and objective evaluation index. The target image synthesized by RD-RaCGAN is similar to the real image and has less noise than the target images generated by the other synthetic methods. As can be seen from the bones of the synthesized brain image and the area near the air interface
the synthetic model in this work visually shows realistic texture details. Compared with the Powell-optimized MI(mutual information) method
ANTs-SyN
D.Demons
Cue-Aware Net(cue-aware deep regression network)
and I-SI(intensity and spatial information) image registration methods
the normalized mutual information increased by 43.71%
12.87%
10.59%
0.47%
and 5.59%
respectively. In addition
the mean square root error decreased by 39.80%
38.67%
15.68%
4.38%
and 2.61%
respectively. The results obtained by the registration algorithm in this study are close to the reference image. The registration effect diagram that the difference between the registration image and the reference image obtained by the algorithm in this study is smaller than that obtained by the other three methods. Small difference between the two images means good registration effect.
Conclusion
2
This study proposes a multimodal brain image registration method based on RD-RaCGAN
which solves the problem of the poor robustness of the model synthesis algorithm based on image synthesis
leading to the inaccuracy of the synthetic image and the poor registration effect.
Arjovsky M, Chintala S and Bottou L. 2017. Wasserstein GAN[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1701.07875.pdf https://arxiv.org/pdf/1701.07875.pdf
Avants B B, Epstein C L, Grossman M and Gee J C. 2008. Symmetric diffeomorphic image registration with cross-correlation:evaluating automated labeling of elderly and neurodegenerative brain. Medical Image Analysis, 12(1):26-41[DOI:10.1016/j.media.2007.06.004]
Bookstein F L. 1989. Principal warps:thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6):567-585[DOI:10.1109/34.24792]
Burgos N, Cardoso M J, Guerreiro F, Veiga C, Modat M, McClelland J, Knopf A C, Punwani S, Atkinson D, Arridge S R, Hutton B F and Ourselin S. 2015. Robust CT synthesis for radiotherapy planning: application to the head and neck region//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer: 476-484[ DOI: 10.1007/978-3-319-24571-3_57 http://dx.doi.org/10.1007/978-3-319-24571-3_57 ]
Buslaev A, Parinov A, Khvedchenya E, Iglovikov V I and Kalinin A A. 2018. Albumentations: fast and flexible image augmentations[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1809.06839v1.pdf https://arxiv.org/pdf/1809.06839v1.pdf
Cao X H, Yang J H, Gao Y Z, Wang Q and Shen D G. 2018a. Region-adaptive deformable registration of CT/MRI pelvic images via learning-based image synthesis. IEEE Transactions on Image Processing, 27(7):3500-3512[DOI:10.1109/TIP.2018.2820424]
Cao X H, Yang J H, Zhang J, Wang Q, Yap P T and Shen D G. 2018b. Deformable image registration using a cue-aware deep regression network. IEEE Transactions on Biomedical Engineering, 65(9):1900-1911[DOI:10.1109/TBME.2018.2822826]
Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: MIT Press: 2672-2680
Han X. 2017. MR-based synthetic CT generation using a deep convolutionalneural network method. Medical Physics, 44(4):1408-1419[DOI:10.1002/mp.12155]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Huang G, Liu Z, van der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 4700-4708[ DOI: 10.1109/CVPR.2017.243 http://dx.doi.org/10.1109/CVPR.2017.243 ]
Huynh T, Gao Y Z, Kang J Y, Wang L, Zhang P, Lian J and Shen D G. 2016. Estimating CT image from MRI data using structured random forest and auto-context model. IEEE Transactions on Medical Imaging, 35(1):174-183[DOI:10.1109/TMI.2015.2461533]
Johnson K A and Becker J A. 1999. The whole brain atlas[EB/OL].[2019-04-10] . http://www.med.harvard.edu/aanlib/home.html http://www.med.harvard.edu/aanlib/home.html
Jolicoeur-Martineau A. 2018. The relativistic discriminator: a key element missing from standard GAN[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1807.00734.pdf https://arxiv.org/pdf/1807.00734.pdf
Kingma D P and Ba J. 2014. Adam: a method for stochastic optimization[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1412.6980.pdf https://arxiv.org/pdf/1412.6980.pdf
Krizhevsky A, Sutskever I and Hinton G E. 2012. ImageNet classification with deep convolutional neural networks//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc: 84-90[ DOI: 10.1145/3065386 http://dx.doi.org/10.1145/3065386 ]
Maes F, Collignon A, Vandermeulen D, Marchal G and Suetens P. 1996. Multi-modality image registration by maximization of mutual information//Proceedings of 1996 Workshop on Mathematical Methods in Biomedical Image Analysis. San Francisco: IEEE: 14-22[ DOI: 10.1109/MMBIA.1996.534053 http://dx.doi.org/10.1109/MMBIA.1996.534053 ]
Mao X D, Li Q, Xie H R, Lau R Y K and Wang Z. 2016. Multi-class generative adversarial networks with the L2 loss function[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1611.04076v1.pdf https://arxiv.org/pdf/1611.04076v1.pdf
Mirza M and Osindero S. 2014. Conditional generative adversarial nets[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1411.1784.pdf https://arxiv.org/pdf/1411.1784.pdf
Nie D, Trullo R, Lian J, Wang L, Petitjean C, Ruan S, Wang Q and Shen D G. 2018. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 65(12):2720-2730[DOI:10.1109/TBME.2018.2814538]
Öfverstedt J, Lindblad J and Sladoje N. 2019. Fast and robust symmetric image registration based on distances combining intensity and spatial information. IEEE Transactions on Image Processing, 28(7):3584-3597[DOI:10.1109/TIP.2019.2899947]
Oliveira F P M and Tavares J M R S. 2014. Medical image registration:a review. Computer Methods in Biomechanics and Biomedical Engineering, 17(2):73-93[DOI:10.1080/10255842.2012.670855]
Ratliff L J, Burden S A and Sastry S S. 2013. Characterization and computation of local Nash equilibria in continuous games//Proceedings of the 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton). Monticello: IEEE: 917-924[ DOI: 10.1109/Allerton.2013.6736623 http://dx.doi.org/10.1109/Allerton.2013.6736623 ]
Roy S, Carass A, Shiee N, Pham D L and Prince J L. 2010. MR contrast synthesis for lesion segmentation//Proceedings of 2010 IEEE International Symposium on Biomedical Imaging: from Nano to Macro. Rotterdam: IEEE: 932-935[ DOI: 10.1109/ISBI.2010.5490140 http://dx.doi.org/10.1109/ISBI.2010.5490140 ]
Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A and Chen X. 2016. Improved techniques for training GANs[EB/OL].[2019-04-10] . https://arxiv.org/pdf/1606.03498.pdf https://arxiv.org/pdf/1606.03498.pdf
Sotiras A, Davatzikos C and Paragios N. 2013. Deformable medical image registration:a survey. IEEE Transactions on Medical Imaging, 32(7):1153-1190[DOI:10.1109/TMI.2013.2265603]
Tai Y, Yang J, Liu X M and Xu C Y. 2017. MemNet: a persistent memory network for image restoration//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 4549-4557[ DOI: 10.1109/ICCV.2017.486 http://dx.doi.org/10.1109/ICCV.2017.486 ]
Tustison N J, Avants B B, Cook P A, Zheng Y J, Egan A, Yushkevich P A and Gee J C. 2010. N4ITK:improved N3 bias correction. IEEE Transactions on Medical Imaging, 29(6):1310-1320[DOI:10.1109/TMI.2010.2046908]
Vercauteren T, Pennec X, Perchant A and Ayache N. 2009. Diffeomorphic demons:efficient non-parametric image registration. NeuroImage, 45(S1):S61-S72[DOI:10.1016/j.neuroimage.2008.10.040]
Xie D, Cao X H, Gao Y Z, Wang L and Shen D G. 2016. Estimating CT image from MRI data using 3D fully convolutional networks//Proceedings of the 1st International Workshop, LABELS 2016, and Second International Workshop on Deep Learning and Data Labeling for Medical Applications. Athens: Springer: 170-178[ DOI: 10.1007/978-3-319-46976-8_18 http://dx.doi.org/10.1007/978-3-319-46976-8_18 ]
Ye C Y, Yang Z, Ying S H and Prince J L. 2015. Segmentation of the cerebellar peduncles using a random forest classifier and a multi-object geometric deformable model:application to spinocerebellar ataxia type 6. Neuroinformatics, 13(3):367-381[DOI:10.1007/s12021-015-9264-7]
Ye D H, Zikic D, Glocker B, Criminisi A and Konukoglu E. 2013. Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization//Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention. Nagoya: Springer: 606-613[ DOI: 10.1007/978-3-642-40811-3_76 http://dx.doi.org/10.1007/978-3-642-40811-3_76 ]
Zhang Y L, Tian Y P, Kong Y, Zhong B N and Fu Y. 2018. Residual dense network for image super-resolution//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE: 2472-2481[ DOI: 10.1109/CVPR.2018.00262 http://dx.doi.org/10.1109/CVPR.2018.00262 ]
相关文章
相关作者
相关机构
京公网安备11010802024621
