Multi-modal MR image super-resolution with residual dense attention network

Yu Liu; Wenyu Zhu; Juan Cheng; Xun Chen

doi:10.11834/jig.220286

Medical Image Processing | Views : 0 下载量: 260 CSCD: 1

PDF
Export
Share
Collection
Album

Multi-modal MR image super-resolution with residual dense attention network
Vol. 28, Issue 1, Pages: 248-259(2023)
Received：23 March 2022，

Revised：2022-4-22，

Accepted：29 April 2022，

Published：16 January 2023
DOI： 10.11834/jig.220286
稿件说明：

移动端阅览

DOI：

Yu Liu, Wenyu Zhu, Juan Cheng, Xun Chen. Multi-modal MR image super-resolution with residual dense attention network[J]. Journal of Image and Graphics, 2023, 28(1): 248-259. DOI： 10.11834/jig.220286.

摘要

目的

现有医学图像超分辨率方法主要针对单一模态图像进行设计，然而在磁共振成像(magnetic resonance imaging，MRI)技术的诸多应用场合，往往需要采集不同成像参数下的多模态图像。针对单一模态的方法无法利用不同模态图像之间的关联信息，很大程度上限制了重建性能。目前超分辨率网络模型参数量往往较大，导致计算和存储代价较高。为此，本文提出了一个轻量级残差密集注意力网络，以一个统一的网络模型同时实现多模态MR图像的超分辨率重建。

方法

首先将不同模态的MR图像堆叠后输入网络，在低分辨率空间中提取共有特征，之后采用设计的残差密集注意力模块进一步精炼特征，再通过一个亚像素卷积层上采样到高分辨率空间，最终分别重建出不同模态的高分辨率图像。

结果

本文采用MICCAI (medical image computing and computer assisted intervention) BraTS (brain tumor segmentation) 2019数据集中的T1和T2加权MR图像对网络进行训练和测试，并与8种代表性超分辨率方法进行对比。实验结果表明，本文方法可以取得优于对比方法的重建性能。相对于对比方法中性能最优的方法，本文方法在参数量低于对比模型的10%的情况下，当放大因子为2时，T1和T2加权模态的峰值信噪比(peak signal to noise ratio，PSNR)分别提高了0.109 8 dB和0.415 5 dB；当放大因子为3时，T2加权模态的PSNR提高了0.295 9 dB，T1加权模态下降了0.064 6 dB；当放大因子为4时，T1和T2加权模态的PSNR分别提高了0.269 3 dB和0.042 9 dB。

结论

本文提出的轻量级超分辨率网络能够充分利用不同模态MR图像之间的关联信息，同时获得各输入模态图像高质量的超分辨率重建结果。

Abstract

Objective

Image super-resolution (SR) is a sort of procedure that aims to reconstruct high resolution (HR) images from a given single or a set of low resolution (LR) images. This cost effective medical-related technique can improve the spatial resolution of images in terms of image processing algorithms. However

most of medical image super-resolution methods are focused on a single-modal super-resolution design. Current magnetic resonance imaging based (MRI-based) clinical applications

Multiple modalities are obtained by different parameter settings. In this case

a single modality super-resolution method cannot take advantage of the correlation information amongst multiple modalities

which limits the super-resolution capability. In addition

most existing deep learning based (DL-based) super-resolution models have constrained of a number of trainable parameters

higher computational cost and memory storage in practice. To strengthen multi-modalities correlation information for reconstruction

our research is focused on a lightweight DL model (i.e.

residual dense attention network) for multi-modal MR image super-resolution.

Method

A residual dense attention network is developed for multi-modal MR image super-resolution. Our network is composed of three parts: 1) shallow feature extraction

2) feature refinement and 3) image reconstruction. Two of multi-modal MR images are input to the network after stacking. First

a 3 × 3 convolutional layer in the shallow feature extraction part is used to extract the initial feature maps in the low-resolution space. Next

the feature refinement part is mainly composed of several residual dense attention blocks. Each residual dense attention block consists of a residual dense block and an efficient channel attention module. Third

dense connection and local residual learning are adopted to improve the representation capability of the network in the residual dense block. The efficient channel attention module is facilitated the network to adaptively identify the feature maps that are more crucial for reconstruction. The outputs of all residual dense attention modules are stacked together and fed into two convolutional layers. These convolution layers are employed to reduce the channels of the feature maps as well as for feature fusion. After that

a global residual learning strategy is implemented to optimize the information flow further. The initial feature maps are added to the last layer through a skip connection. Finally

the obtained low-resolution feature maps in the image reconstruction part are up-scaled to the high-resolution space by a sub-pixel convolutional layer. Additionally

two symmetric branches are used to reconstruct the super-resolution results of the different modalities. To reconstruct the residual maps of the two modalities

each branch of them consists of two 3 × 3 convolutional layers. To obtain the final super-resolution results

the residual maps are linked to the interpolated low-resolution images. The popular L1 loss is used to optimize the network parameters.

Result

In the experiments

to verify the effectiveness of the proposed method

the MR images of two modalities (i.e.

T1-weighted and T2-weighted) from the medical image computing and computer assisted intervention (MICCAI) brain tumor segmentation (BraTS) 2019 are adopted. The original MRI scans are split and segmented into a training set

a validation set and a testing set. To verify the effect of the multi-modal image super-resolution manner and the efficient channel attention module

two sets of ablation experiments are designed. The results show that these two of components can optimize the super-resolution performance more. Furthermore

eight representative image super-resolution methods are used for comparative analysis of performance in the experiments. Experimental results demonstrate that our method can improve these reference methods in terms of both of the objective evaluation and visual quality. Specifically

our method can obtain more competitive results as mentioned below: 1) when the up-scale factor is 2

the peak signal to noise ratio (PSNR) of the T1-weighted and T2-weighted modalities improve by 0.109 8 dB and 0.415 5 dB

respectively; 2) when the up-scale factor is 3

the PSNR of the T2-weighted modality improves by 0.295 9 dB while the T1-weight modality decreases by 0.064 6 dB; 3) when the up-scale factor is 4

the PSNR of the T1-weighted and T2-weighted modalities improve by 0.269 3 dB and 0.042 9 dB

respectively. It is worth noting that our network has a more than 10 times reduction in terms of network parameters compared to the popular reference method.

Conclusion

The correlation information of different modalities between MR images is beneficial to image super-resolution. Our multi-modal MR image super-resolution method can achieve high-quality super-resolution results of two modalities simultaneously in an integrated correlation information-based network. It can obtain more competitive performance than the state-of-the-art super-resolution methods with a relative lightweight model.

关键词

Keywords

references

Chen L H, Yang X M, Jeon G, Anisetti M and Liu K. 2020. A trusted medical image super-resolution method based on feedback adaptive weighted dense network. Artificial Intelligence in Medicine, 106: #101857 [DOI: 10.1016/j.artmed.2020.101857]

Cheng D Q, Guo X, Chen L L, Kou Q Q, Zhao K and Gao R. 2021. Image super-resolution reconstruction from multi-channel recursive residual network. Journal of Image and Graphics, 26(3): 605-618

程德强, 郭昕, 陈亮亮, 寇旗旗, 赵凯, 高蕊. 2021. 多通道递归残差网络的图像超分辨率重建. 中国图象图形学报, 26(3): 605-618 [DOI: 10.11834/jig.200108]

Dong C, Loy C C, He K M and Tang X O. 2014. Learning a deep convolutional network for image super-resolution//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 184-199 [ DOI: 10.1007/978-3-319-10593-2_13 http://dx.doi.org/10.1007/978-3-319-10593-2_13 ]

Dong C, Loy C C and Tang X O. 2016. Accelerating the super-resoluti on convolutional neural network//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 391-407 [ DOI: 10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25 ]

Kim J, Lee J K and Lee K M. 2016a. Accurate image super-resolution using very deep convolutional networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 1646-1654 [ DOI: 10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ]

Kim J, Lee J K and Lee K M. 2016b. Deeply-recursive convolutional network for image super-resolution//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 1637-1645 [ DOI: 10.1109/CVPR.2016.181 http://dx.doi.org/10.1109/CVPR.2016.181 ]

Lei P C, Liu C, Tang J G and Peng D L. 2020. Hierarchical feature fusion attention network for image super-resolution reconstruction. Journal of Image and Graphics, 25(9): 1773-1786

雷鹏程, 刘丛, 唐坚刚, 彭敦陆. 2020. 分层特征融合注意力网络图像超分辨率重建. 中国图象图形学报, 25(9): 1773-1786 [DOI: 10.11834/jig.190607]

Li Z, Yang J L, Liu Z, Yang X M, Jeon G and Wu W. 2019. Feedback network for image super-resolution//Proce edings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 3862-3871 [ DOI: 10.1109/CVPR.2019.00399 http://dx.doi.org/10.1109/CVPR.2019.00399 ]

Lim B, Son S, Kim H, Nah S and Lee K M. 2017. Enhanced deep residual networks for single image super-resolution//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern recognition workshops (CVPRW). Honolulu, USA: IEEE: 1132-1140 [ DOI: 10.1109/CVPRW.2017.151 http://dx.doi.org/10.1109/CVPRW.2017.151 ]

Shen M Y, Yu P F, Wang R G, Yang J and Xue L X. 2019. Image super-resolution reconstruction via deep network based on multi-staged fusion. Journal of Image and Graphics, 24(8): 1258-1269

沈明玉, 俞鹏飞, 汪荣贵, 杨娟, 薛丽霞. 2019. 多阶段融合网络的图像超分辨率重建. 中国图象图形学报, 24(8): 1258-269 [DOI: 10.11834/jig.180619]

Shi F, Cheng J, Wang L, Yap P T and Shen D G. 2015. LRTV: MR image super-resolution with low-rank and total variation regularizations. IEEE Transactions on Medical Imaging, 34(12): 2459-2466 [DOI: 10.1109/TMI.2015.2437894]

Shi J, Li Z, Ying S H, Wang C F, Liu Q P, Zhang Q and Yan P K. 2019. MR image super-resolution via wide residual networks with fixed skip connection. IEEE Journal of Biomedical and Health Informatics, 23(3): 1129-1140 [DOI: 10.1109/JBHI.2018.2843819]

Shi W Z, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D and Wang Z H. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 1874-1883 [ DOI: 10.1109/CVPR.2016.207 http://dx.doi.org/10.1109/CVPR.2016.207 ]

Tian C W, Xu Y, Zuo W M, Zhang B, Fei L K and Lin C W. 2021. Coarse-to-fine CNN for image super-resolution. IEEE Transactions on Multimedia, 23: 1489-1502 [DOI: 10.1109/TMM.2020.2999182]

Tong J C, Fei J L, Chen J S, Li H and Ding D D. 2019. Multi-level feature fusion image super-resolution algorithm with recursive neural network. Journal of Image and Graphics, 24(2): 302-312

佟骏超, 费加罗, 陈靖森, 李恒, 丁丹丹. 2019. 递归式多阶特征融合图像超分辨率算法. 中国图象图形学报, 24(2): 302-312 [DOI: 10.11834/jig.180410]

Tong T, Li G, Liu X J and Gao Q Q. 2017. Image super-resolution using dense skip connections//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 4809-4817 [ DOI: 10.1109/ICCV.2017.514 http://dx.doi.org/10.1109/ICCV.2017.514 ]

Wang Q L, Wu B G, Zhu P F, Li P H, Zuo W M and Hu Q H. 2020. ECA-Net: efficient channel attention for deep convolutional neural networks [EB/OL ] . [2022-01-18 ] . https://arxiv.org/pdf/1910.03151v3.pdf https://arxiv.org/pdf/1910.03151v3.pdf

Xue X T, Wang Y, Li J, Jiao Z C, Ren Z Q and Gao X B. 2020. Progressive sub-band residual-learning network for MR image super resolution. IEEE Journal of Biomedical and Health Informatics, 24(2): 377-386 [DOI: 10.1109/JBHI.2019.2945373]

Ying Z L and Long X. 2019. Single-image super-resolution construction based on multi-scale dense residual network. Journal of Image and Graphics, 24(3): 410-419

应自炉, 龙祥. 2019. 多尺度密集残差网络的单幅图像超分辨率重建. 中国图象图形学报, 24(3): 410-419 [DOI: 10.11834/jig.180431]

Zeng K, Zheng H, Cai C B, Yang Y, Zhang K H and Chen Z. 2018. Simultaneous single- and multi-contrast super-resolution for brain MRI images based on a convolutional neural network. Computers in Biology and Medicine, 99: 133-141 [DOI: 10.1016/j.compbiomed.2018.06.010]

Zhang Y L, Tian Y P, Kong Y, Zhong B N and Fu Y. 2018. Residual dense network for image super-resolution//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt La ke City, USA: IEEE: 2472-2481 [ DOI: 10.1109/CVPR.2018.00262 http://dx.doi.org/10.1109/CVPR.2018.00262 ]

Zhou B, Li C H and Chen W. 2021. Region-level channel attention for single image super-resolution combining high frequency loss. Journal of Image and Graphics, 26(12): 2836-2847

周波, 李成华, 陈伟. 2021. 区域级通道注意力融合高频损失的图像超分辨率重建. 中国图象图形学报, 26(12): 2836-2847 [DOI: 10.11834/jig.200582]