DeepLabv3plus-IRCNet: an image semantic segmentation method for small target feature extraction

Wen Liu; Hairong Wang; Beijing Zhou

doi:10.11834/jig.190576

lmage Analysis and Recognition | Views : 0 下载量: 176 CSCD: 6

PDF
Export
Share
Collection
Album

DeepLabv3plus-IRCNet: an image semantic segmentation method for small target feature extraction
Vol. 26, Issue 2, Pages: 391-401(2021)
Received：11 November 2019，

Revised：2020-6-5，

Accepted：12 June 2020，

Published：16 February 2021
DOI： 10.11834/jig.190576
稿件说明：

移动端阅览

DOI：

Wen Liu, Hairong Wang, Beijing Zhou. DeepLabv3plus-IRCNet: an image semantic segmentation method for small target feature extraction[J]. Journal of Image and Graphics, 2021, 26(2): 391-401. DOI： 10.11834/jig.190576.

摘要

目的

为了解决经典卷积神经网络无法满足图像中极小目标特征提取的准确性需求问题，本文基于DeepLabv3plus算法，在下采样过程中引入特征图切分模块，提出了DeepLabv3plus-IRCNet（IR为倒置残差（inverted residual，C为特征图切分（feature map cut））图像语义分割方法，支撑图像极小目标的特征提取。

方法

采用由普通卷积层和多个使用深度可分离卷积的倒置残差模块串联组成的深度卷积神经网络提取特征，当特征图分辨率降低到输入图像的1/16时，引入特征图切分模块，将各个切分特征图分别放大，通过参数共享的方式提取特征。然后，将每个输出的特征图进行对应位置拼接，与解码阶段放大到相同尺寸的特征图进行融合，提高模型对小目标物体特征的提取能力。

结果

本文方法引入特征图切分模块，提高了模型对小目标物体的关注，充分考虑了图像上下文信息，对多个尺度下的各个中间层特征进行融合，提高了图像分割精度。为验证方法的有效性，使用CamVid（Cambridge-driving labeled video database）数据集对提出的方法进行验证，平均交并比（mean intersection over union，mIoU）相对于DeepLabv3plus模型有所提升。验证结果表明了本文方法的有效性。

结论

本文方法充分考虑了图像分割中小目标物体的关注度，提出的DeepLabv3plus-IRCNet模型提升了图像分割精度。

Abstract

Objective

A huge amount of image data have been generated with the development of the Internet of things and artificial intelligence technology and their widespread application to various fields. Understanding image content quickly and accurately and automatically segmenting the target area of an image in accordance with the requirements of the application scene have become the focus of many researchers. In recent years

image semantic segmentation methods based on deep learning have been developed steadily. These methods have been widely used in automatic driving and robot engineering

and have become the primary research task in computer vision. Common convolutional neural networks (CNNs) can efficiently extract the features of an image. They typically operate directly on the entire feature map. However

extremely small targets frequently occur in a local area of an image. The common convolution operation cannot efficiently extract the features of small targets. To solve this problem

the feature image cut module is introduced into the down-sampling process.

Method

At present

the spatial pyramid pool module and codec structure of a deep CNN (DCNN) have become the mainstream method for image semantic segmentation. The former network can extract the features of an input feature map by using filters or pooling operations with multiple rates and effective fields

and thus

encode the multi-scale context information. Meanwhile

the latter network can capture clearer object boundaries by gradually recovering spatial information. However

many difficulties and challenges persist. The first problem is that the DCNN model has extremely high requirements for the hardware platform and is unsuitable for real-time engineering applications. The second problem is that the resolution of the feature image shrinks after the image is encoded

resulting in the loss of the spatial information of some pixels. The third problem is that the segmentation process cannot effectively consider the image context information (i.e.

the relationship among pixels) and cannot fully utilize rich spatial location information. The fourth problem is that DCNNs are not good at capturing feature expression

and thus

achieving a better semantic segmentation effect is difficult. To solve these problems

this study proposes an improved image semantic segmentation algorithm DeepLab IRCNet based on DeepLabv3+ to solve the problem in which DCNNs experience difficulty in extracting the features of small and medium-sized objects. In the encoder part

a DCNN composed of a series of ordinary convolutional layers and multiple inverted residual modules is used to extract features. In the inverted residual module

deep separable convolutions are used instead of ordinary convolutions. When the resolution of the feature image is reduced to 1/16 of the input image

the feature map is divided equally

the feature map after segmentation is enlarged to the size before segmentation

and the feature extraction module is used to share each segmented feature map through parameter sharing. Consequently

the model can focus better on small target objects in the local area after feature segmentation. On the main network

the extracted feature map is continuously inputted into the hollow space pyramid pooling module to capture the multi-scale contextual content information of the image

and the hollow convolution with a void rate of {6

18} in the atrous spatial pyramid pooling module is used. The sequence also parallels a 1×1 convolutional layer and image pooling

wherein the choice of the void rate is the same as that of DeepLabv3+

improving segmentation performance. Then

a 1×1 convolution is used to obtain the output tensor of the target feature map. In the decoder part

bilinear interpolation is used to up-sample two times

and then the up sampling feature mapis fused with the output feature map of the feature segmentation module in the encoder. Several 3×3 depth separable convolutions are used to redefine the feature

and bilinear interpolation is used for up-sampling. Finally

an image semantic segmentation map that is the same size as the input image is the output.

Result

In this study

the CamVid(Cambridege-driving labeled video database) dataset is used to verify the proposed method. The mean intersection over union(mIoU) is increased by 1.5 percentage points compared with the DeepLabv3+ model. The verification results show the effectiveness of the proposed method.

Conclusion

In this study

a feature graph segmentation module is introduced to improve model attention to small objects and address the problem of low semantic segmentation accuracy.

关键词

Keywords

references

Badrinarayanan V, Kendall A and Cipolla R. 2017. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12): 2481-2495[DOI: 10.1109/TPAMI.2016.2644615]

Brostow G J, Fauqueur J and Cipolla R. 2009. Semantic object classes in video: a high-definition ground truth database. Pattern Recognition Letters, 30(2): 88-97[DOI: 10.1016/j.patrec.2008.04.005]

Chen L C, Papandreou G, Kokkinos I, Murphy K and Yuille A L. 2014. Semantic image segmentation with deep convolutional nets and fully connected CRFs[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1412.7062.pdf https://arxiv.org/pdf/1412.7062.pdf

Chen L C, Papandreou G, Kokkinos I, Murphy K and Yuille A L. 2018a. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834-848[DOI: 10.1109/TPAMI.2017.2699184]

Chen L C, Papandreou G, Schroff F and Adam H. 2017. Rethinking atrous convolution for semantic image segmentation[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1706.05587v3.pdf https://arxiv.org/pdf/1706.05587v3.pdf

Chen L C, Zhu Y K, Papandreou G, Schroff F and Adam H. 2018b. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 833-851[ DOI: 10.1007/978-3-030-01234-2_49 http://dx.doi.org/10.1007/978-3-030-01234-2_49 ]

Chollet F. 2017. Xception: deep learning with depthwise separable convolutions//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 1800-1807[ DOI: 10.1109/CVPR.2017.195 http://dx.doi.org/10.1109/CVPR.2017.195 ]

Dai J F, Qi H Z, Xiong Y W, Li Y, Zhang G D, Hu H and Wei Y C. 2017. Deformable convolutional networks//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 764-773[ DOI: 10.1109/ICCV.2017.89 http://dx.doi.org/10.1109/ICCV.2017.89 ]

Fu C Y, Liu W, Ranga A, Tyagi A and Berg A C. 2017. DSSD: deconvolutional single shot detector[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1701.06659.pdf https://arxiv.org/pdf/1701.06659.pdf

He K M, Zhang X Y, Ren S Q and Sun J. 2015. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9): 1904-1916[DOI: 10.1109/TPAMI.2015.2389824]

Howard A, Sandler M, Chu G, Chen L C, Chen B, Tan M X, Wang W J, Zhu Y K, Pang R M, Vasudevan V, Le Q V and Adam H. 2019. Searching for mobilenetv3[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1905.02244.pdf https://arxiv.org/pdf/1905.02244.pdf

Howard A G, Zhu M L, Chen B, Kalenichenko D, Wang W J, Weyand T, Andreetto M and Adam H. 2017. Mobilenets: efficient convolutional neural networks for mobile vision applications[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1704.04861.pdf https://arxiv.org/pdf/1704.04861.pdf

Lin T Y, Dollár P, Girshick R, He K M, Hariharan B and Belongie S. 2017. Feature pyramid networks for object detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 936-944[ DOI: 10.1109/CVPR.2017.106 http://dx.doi.org/10.1109/CVPR.2017.106 ]

Long J, Shelhamer E and Darrell T. 2015. Fully convolutional networks for semantic segmentation//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 3431-3440[ DOI: 10.1109/CVPR.2015.7298965 http://dx.doi.org/10.1109/CVPR.2015.7298965 ]

Ma N N, Zhang X Y, Zheng H T and Sun J. 2018. ShuffleNet v2: practical guidelines forefficient CNN architecture design//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 122-138[ DOI: 10.1007/978-3-030-01264-9_8 http://dx.doi.org/10.1007/978-3-030-01264-9_8 ]

Newell A, Yang K Y and Deng J. 2016. Stacked hourglass networks for human pose estimation//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 483-499[ DOI: 10.1007/978-3-319-46484-8_29 http://dx.doi.org/10.1007/978-3-319-46484-8_29 ]

Peng C, Zhang X Y, Yu G, Luo G M and Sun J. 2017. Large kernel matters—improve semantic segmentation by global convolutional network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 1743-1751[ DOI: 10.1109/CVPR.2017.189 http://dx.doi.org/10.1109/CVPR.2017.189 ]

Ronneberger O, Fischer P and Brox T. 2015. U-net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241[ DOI: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]

Sandler M, Howard A, Zhu M L, Zhmoginov A and Chen L C. 2018. Mobilenetv2: inverted residuals and linear bottlenecks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 4510-4520[ DOI: 10.1109/CVPR.2018.00474 http://dx.doi.org/10.1109/CVPR.2018.00474 ]

SimonyanK and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf

Wang P Q, Chen P F, Yuan Y, Liu D, Huang Z H, Hou X D and Cottrell G. 2018. Understanding convolution for semantic segmentation//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, USA: IEEE: 1451-1460[ DOI: 10.1109/WACV.2018.00163 http://dx.doi.org/10.1109/WACV.2018.00163 ]

Yu F and Koltun V. 2015. Multi-scale context aggregation by dilated convolutions[EB/OL].[2019-10-27] . https://arxiv.org/pdf/1511.07122.pdf https://arxiv.org/pdf/1511.07122.pdf

Zhang X Y, Zhou X Y, Lin M X and Sun J. 2018. Shufflenet: an extremely efficient convolutional neural network for mobile devices//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 6848-6856[ DOI: 10.1109/CVPR.2018.00716 http://dx.doi.org/10.1109/CVPR.2018.00716 ]

Zhao H S, Qi X J, Shen X Y, Shi J P and Jia J Y. 2018. ICNet for real-time semantic segmentation on high-resolution images//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 418-434[ DOI: 10.1007/978-3-030-01219-9_25 http://dx.doi.org/10.1007/978-3-030-01219-9_25 ]

Zhao H S, Shi J P, Qi X J, Wang X G and Jia J Y. 2017. Pyramid scene parsing network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE:6230-6239[ DOI: 10.1109/CVPR.2017.660 http://dx.doi.org/10.1109/CVPR.2017.660 ]

Alert me when the article has been cited

提交

Single-stage object detection using filter pyramid and atrous convolution

Cross-modal feature fusion and detail-enhanced RGB-D salient object detection

Two-stage vision transformer for fusing global and local features in distracted driving behavior recognition

MoLiNet： a local and global information interactive fusion network for improving multi-classification of pathological image artifacts

Image denoising via Swin Transformer V2 and feature fusion U-Net