A spatio-temporal multi-scale network for cardiac dynamic MRI image segmentation

Jiachen Xu; Zhiyong Xiao

doi:10.11834/jig.200783

Magnetic Resonance Image | Views : 0 下载量: 166 CSCD: 2

PDF
Export
Share
Collection
Album

A spatio-temporal multi-scale network for cardiac dynamic MRI image segmentation
Vol. 27, Issue 3, Pages: 862-872(2022)
Received：21 December 2020，

Revised：2021-4-22，

Accepted：29 April 2021，

Published：16 March 2022
DOI： 10.11834/jig.200783
稿件说明：

移动端阅览

DOI：

Jiachen Xu, Zhiyong Xiao. A spatio-temporal multi-scale network for cardiac dynamic MRI image segmentation[J]. Journal of Image and Graphics, 2022, 27(3): 862-872. DOI： 10.11834/jig.200783.

摘要

目的

心脏磁共振成像(magnetic resonance imaging

MRI)的自动分割技术有利于在临床诊断中评估心脏的功能参数。然而由于心脏磁共振成像技术产生的图像边界不清晰、各向分辨率异性等特性，现有的大多数方法依旧存在类内不确定、类间不清晰问题。针对这一问题，提出了一种利用时间信息进行特征增强，并利用空间信息进行特征矫正的多输入、多分支和多任务的分割网络(spatio-temporal UNet，ST UNet)。

方法

为充分获取动态心脏MRI图像的时间信息，提出了全新的时间增强编码模块，将需要进行分割的目标帧和一段包含了目标帧的连续时间片段作为关键序列一同输入网络。关键序列用于获取更丰富的时间特征，目标帧提供更精准的边缘特征。为了聚集更多有益的特征，更好地融合时域特征和边缘特征，采用可变形全局连接代替传统的长连接，为网络的解码部分提供更广泛的多维特征信息。在训练过程中额外学习空间方向场特征，并使用该特征对原有的分割结果进行矫正。

结果

在ACDC(Automated Cardiac Diagnosis Challenge)心脏分割挑战中，以${\rm{Dice}}$系数和${\rm{HD}}$(Hausdorff distance)距离为评价指标，该方法在左心室、右心室和左心肌分割的平均${\rm{Dice}}$系数分别为95%、91.5%和91%，${\rm{HD}}$距离的平均值分别为6.77、11.39和8.54。

结论

实验表明，提出的新型网络能够充分地利用心脏MRI图像的时空信息，有效地提升目标器官的分割效果，更有助于医生对于心脏诊断。

Abstract

Objective

The segmentation of cardiac dynamic magnetic resonance imaging (MRI) is essential to evaluate cardiac functional parameters in clinical diagnosis. Based on the segmentation results

the qualified analysis can be obtained the myocardial mass and thickness

ejection fraction

ventricular volume and other diagnostic indicators effectively. Currently

the method of heart segmentation is still limited to manual segmentation. This method is time-consuming and human behavior-oriented. Therefore

the issue of automatic and accurate cardiac-MRI (CMRI) segmentation has been focused on. However

due to the non-uniform magnetic field intensity

artifacts resulted blurred boundaries in the process of imaging. Organs of different subjects vary greatly

especially the variable shape and size of the right ventricle

which tends to produce volume effect. In addition

dynamic magnetic resonance imaging is featured based on a small scale of short axis sections with thick slice

which results in low short axis resolution and sparse information of the image. As the existing data sets only including ground truth of dynamic MRI images at two sites in the end of systole and end of diastole

the existing networks usually only take the images at these two sites as segmentation objects

which ignores the information of dynamic MRI images timescale. Hence

automatic segmentation of dynamic cardiac MRI images is challenged of the issues of intra-class uncertainty and inter-class imbalance. This illustration demonstrates a spatio-temporal multi-scale UNet that makes use of time information to conduct feature enhancement and spatial information to get feature correction.

Method

First

a new time-enhanced coding path is developed in order to fully obtain the time information of dynamic CMRI images

which consists of two branches are those of the target frame branch and the key sequence branch. The target frame is the image to be segmented

and the key sequence is the consistent time series containing the target frame. The key sequence is used to obtain richer time features

while the target frame provides more accurate edge features. Due to the beating frequency of the heart

the boundary information of the target organ would be in conflict with that of the time scaled images in the key sequence

and the edge information extracted from the target frame could make up for this problem. It is worth noting that the key sequence and target frame are cut or filled into 256×256 pixels due to the different sizes of images in the data set. Simultaneously

this analysis also makes the same data enhancement for the target frame and the key sequence and random affine changing and random rotation in order to extend the amount of data in the training set and improve the generalization ability and robustness of the model. Meanwhile

in order to aggregate more beneficial features and better integrate edge information and time features

this research illustrated a deformable global connection instead of the traditional long connection to provide more extensive multi-dimensional feature information for the decoding part of the network. Compared with the original network which only transfers the results of the encoding layer to the decoding layer

the features of the deformable global connection are the weighted sum of the global features. The deformable global connection is composed of the context modeling part

the feature transformation part and the deformable convolutional layer part. The part of modeling algorithm obtains the attention weight of the feature vector in the $x-y$ plane

which represents the global context feature. The feature transformation captures the dependencies between channels to reduce the optimization difficulty via layer normalization. The deformable convolutional layer captures the geometry of the heart. In the process of decoding

the decoding layer can perform targeted aggregation to improve the compact and semantic consistency within the class based on the obtained weights. At the end

the spatial directional field features are learned and the original segmentation results are calibrated by these features. The spatial direction field

generated by ground truth

provides each pixel with a unique path from the boundary to the central region. The orientation relationship is revealed amongst the pixels on the target organ. The feature of central region is used to correct the segmentation map based on the requirement of spatial direction field.

Result

The online test results of the Automated Cardiac Diagnosis Challenge(ACDC) cardiac segmentation challenge illustrated that the average dice for the segmentation of left ventricle

right ventricle and left ventricle in this research analyses were 95%

91.5% and 91% each

which is 2%

1.5% and 2.5% higher than the initial network analysis. The average Hausdorff distance of the segmentation results is 6.77

11.39 and 8.54

respectively

which is 2.22

2.07 and 3.64 lower than the initial network.

Conclusion

This demonstrated experiments show that the proposed network can improve the segmentation effect of target organs effectively based on the spatio-temporal information of cardiac MRI images.

关键词

Keywords

references

Baumgartner C F, Koch L M, Pollefeys M and Konukoglu E. 2018. An exploration of 2D and 3D deep learning techniques for cardiac MR image segmentation//Proceedings of the 8th International Workshop on Statistical Atlases and Computational Models of the Heart. Quebec City, Canada: Springer: 111-119[ DOI: 10.1007/978-3-319-75541-0_12 http://dx.doi.org/10.1007/978-3-319-75541-0_12 ]

Bernard O, Lalande A, Zotti C, Cervenansky F, Yang X, Heng P A, Cetin I, Lekadir K, Camara O, Gonzalez Ballester M A, Sanroma G, Napel S, Petersen S, Tziritas G, Grinias E, Khened M, Kollerathu V A, Krishnamurthi G, Rohe M M, Pennec X, Sermesant M, Isensee F, Jager P, Maier-Hein K H, Full P M, Wolf I, Engelhardt S, Baumgartner C F, Koch L M, Wolterink J M, Isgum I, Jang Y, Hong Y O N M, Patravali J, Jain S, Humbert O and Jodoin P M. 2018. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Transactions on Medical Imaging, 37(11): 2514-2525[DOI: 10.1109/TMI.2018.2837502]

Cao Y, Xu J R, Lin S, Wei F Y and Hu H. 2019. GCNet: non-local networks meet squeeze-excitation networks and beyond//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshops. Seoul, Korea(South): IEEE: 1971-1980[ DOI: 10.1109/ICCVW.2019.00246 http://dx.doi.org/10.1109/ICCVW.2019.00246 ]

Chang Y K, Song B Y, Jung C and Huang L Y. 2018. Automatic segmentation and cardiopathy classification in cardiac MRI images based on deep neural networks//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Calgary, Canada: IEEE: 1020-1024[ DOI: 10.1109/ICASSP.2018.8461261 http://dx.doi.org/10.1109/ICASSP.2018.8461261 ]

Cheng F, Chen C, Wang Y K, Shi H, Cao Y K, Tu D D, Zhang C Z and Xu Y C. 2020. Learning directional feature maps for cardiac MRI segmentation//Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention. Lima, Peru: Springer: 108-117[ DOI: 10.1007/978-3-030-59719-1_11 http://dx.doi.org/10.1007/978-3-030-59719-1_11 ]

Dai J F, Qi H Z, Xiong Y W, Li Y, Zhang G D, Hu H and Wei Y C. 2017. Deformable convolutional networks//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 764-773[ DOI: 10.1109/ICCV.2017.89 http://dx.doi.org/10.1109/ICCV.2017.89 ]

Gao Q, Gao J Y and Zhao D. 2020. GNNIUGnet: Precise Segmentation Neural Network of Left Ventricular Contours for MRI Images Based on Group Normalization and Nearest Interpolation. Computer Science, 47(08): 213-220

高强, 高敬阳, 赵地. 2020. GNNI U-net: 基于组归一化与最近邻插值的MRI左心室轮廓精准分割网络. 计算机科学, 47(08): 213-220 [DOI:10.11896 /jsjkx.190600026]

Hasan S M K and Linte C A. 2020. CondenseUNet: a memory-efficient condensely-connected architecture for bi-ventricular blood pool and myocardium segmentation//Proceedings of SPIE 11315, Medical Imaging 2020: Image-Guided Procedures, Robotic Interventions, and Modeling. Houston, USA: SPIE: 113151 J[ DOI: 10.1117/12.2550640 http://dx.doi.org/10.1117/12.2550640 ]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Liu C, Lin N, Cao Y J and Yang C. 2021. Seg-CapNet: neural network model for the cardiac MRI segmentation. Journal of Image and Graphics, 26(02): 0452-0463

刘畅, 林楠, 曹仰杰, 杨聪. 2021. Seg-CapNet : 心脏MRI图像分割神经网络模型. 中国图象图形学报, 26(02): 0452-0463 [DOI:10.11834/jig.190626]

Liu T, Tian Y, Zhao S F, Huang X Y and Wang Q J. 2020. Residual Convolutional neural network for cardiac image segmentation and heart disease diagnosis. IEEE Access, 8: 82153-82161[DOI: 10.1109/ACCESS.2020.2991424]

Painchaud N, Skandarani Y, Judge T, Bernard O, Lalande A and Jodoin P M. 2019. Cardiac MRI segmentation with strong anatomical guarantees//Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention. Shenzhen:Springer: 632-640[ DOI: 10.1007/978-3-030-32245-8_70 http://dx.doi.org/10.1007/978-3-030-32245-8_70 ]

Redmon J, Divvala S, Girshick R and Farhadi A. 2016. You only look once: unified, real-time object detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 779-788[ DOI: 10.1109/CVPR.2016.91 http://dx.doi.org/10.1109/CVPR.2016.91 ]

Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241[ DOI: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]

Simantiris G and Tziritas G. 2020. Cardiac MRI segmentation with a dilated CNN incorporating domain-specific constraints. IEEE Journal of Selected Topics in Signal Processing, 14(6): 1235-1243[DOI: 10.1109/JSTSP.2020.3013351]

Sun J, Darbeha F, Zaidi M and Wang B. 2020. SAUNet: shape attentive U-Net for interpretable medical image segmentation//Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention. Lima, Peru: Springer: 797-806

Zotti C, Luo Z M, Lalande A and Jodoin P M. 2019. Convolutional neural network with shape prior applied to cardiac MRI segmentation. IEEE Journal of Biomedical and Health Informatics, 23(3): 1119-1128[DOI: 10.1109/JBHI.2018.2865450]

Alert me when the article has been cited

提交

Image colorization via color query and feature enhancement

Foggy ship re-identification network based on multiple feature cascade enhancement and cross-layer adaptive fusion

Hand-object pose estimation method based on fusion feature enhancement and complementary

Ship name text detection method with scene priors fusion

Single-domain generalized breast tumor detection in X-ray images