Segmentation in multi-spectral remote sensing images using the fully convolutional neural network

Jianhua Yao; Jiamin Wu; Yong Yang; Zuxian Shi

doi:10.11834/jig.190157

Remote Sensing Image Processing | Views : 0 下载量: 69 CSCD: 0

PDF
Export
Share
Collection
Album

Segmentation in multi-spectral remote sensing images using the fully convolutional neural network
Vol. 25, Issue 1, Pages: 180-192(2020)
Received：29 April 2019，

Revised：2019-6-10，

Accepted：17 June 2019，

Published：16 January 2020
DOI： 10.11834/jig.190157
稿件说明：

移动端阅览

DOI：

Jianhua Yao, Jiamin Wu, Yong Yang, Zuxian Shi. Segmentation in multi-spectral remote sensing images using the fully convolutional neural network[J]. Journal of Image and Graphics, 2020, 25(1): 180-192. DOI： 10.11834/jig.190157.

摘要

目的

传统的遥感影像分割方法需要大量人工参与特征选取以及参数选择，同时浅层的机器学习算法无法取得高精度的分割结果。因此，利用卷积神经网络能够自动学习特征的特性，借鉴处理自然图像语义分割的优秀网络结构，针对遥感数据集的特点提出新的基于全卷积神经网络的遥感影像分割方法。

方法

针对遥感影像中目标排列紧凑、尺寸变化大的特点，提出基于金字塔池化和DUC（dense upsampling convolution）结构的全卷积神经网络。该网络结构使用改进的DenseNet作为基础网络提取影像特征，使用空间金字塔池化结构获取上下文信息，使用DUC结构进行上采样以恢复细节信息。在数据处理阶段，结合遥感知识将波段融合生成多源数据，生成植被指数和归一化水指数，增加特征。针对遥感影像尺寸较大、采用普通预测方法会出现拼接痕迹的问题，提出基于集成学习的滑动步长预测方法，对每个像素预测14次，每次预测像素都位于不同图像块的不同位置，对多次预测得到的结果进行投票。在预测结束后，使用全连接条件随机场（CRFs）对预测结果进行后处理，细化地物边界，优化分割结果。

结果

结合遥感知识将波段融合生成多源数据可使分割精度提高3.19%；采用基于集成学习的滑动步长预测方法可使分割精度较不使用该方法时提高1.44%；使用全连接CRFs对预测结果进行后处理可使分割精度提高1.03%。

结论

针对宁夏特殊地形的遥感影像语义分割问题，提出基于全卷积神经网络的新的网络结构，在此基础上采用集成学习的滑动步长预测方法，使用全连接条件随机场进行影像后处理可优化分割结果，提高遥感影像语义分割精度。

Abstract

Objective

The traditional remote sensing image segmentation method requires the selection of manyartificial participation featuresandparameters. The shallow machine learning algorithm cannot achieve high-precision segmentation accuracy. The convolutional neural network can automatically learn the characteristics of features and draws on its excellent network structure for performing natural image semantic segmentation. A novel method based on the fully convolutional neural network for remote sensing image segmentation is proposed based on the characteristics of the remote sensing dataset. It studies the fusion between multi-spectral image data bands

increases the learnable features

and improves segmentation accuracy. On the basis of the characteristics of the remote sensing image size

the prediction results of integrated learning and the conditional random field processing model are investigated to mitigate the phenomenon of misclassification

restore the boundary of features

and further improve segmentation accuracy. This study realizes the extraction of features on multi-spectral remote sensing images

which can be applied to subsequent change detection tasks

thus promoting the analysis of changes in surface cover types by automation.

Method

Aiming at the characteristics of compact targets and the large size range of remote sensing images

a fully convolutional neural network based on pyramid pooling and the dense upsamplingconvolution (DUC) structure is proposed. The proposed network can automatically interpret remote sensing images. The network structure uses improved DenseNet as the underlying network to extract image features

the spatial pyramid pooling structure to obtain context information

and the DUC structure to upsampleand recover detailed information. In the data processing stage

in combination with remote sensing knowledge

the bands are combined to generate multi-source data

and vegetation and normalized water indexes are generated to increase the characteristics. A sliding step prediction method based on integrated learning is proposed to address the problem of remote sensing images being large and the appearance of splicing trace by an ordinary prediction method. Each pixel is predicted 1 to 4 times

and each predicted pixel is located in different image blocks. Different locations vote on the results of multiple predictions. After prediction

the prediction results are post-processed using fully connected conditional random fields (CRFs) to refine the boundary of the features and optimize the segmentation results.

Result

To verify the validity of the proposed network model and post-processing method

the U-Net model

the fully convolutional neural network FCN-8s model

and the Hdc-DUC model are compared through experiments using a self-built dataset. The accuracy of using the multi-source data from the training model is higher than that obtained by using the original data. The multi-source data training model improves the mIoU evaluation standard by 3.19%

which confirms the validity of the multi-source data generated by band fusion combined with geo-remote sensing knowledge. In terms of effectiveness

when the sliding step prediction method based on integrated learning is used

the segmentation accuracy is improved by 1.44%

and the effect of the characteristics of the remote sensing image on the prediction phase of the model is verified. Although fully connected CRFs may smoothen small-sized features

the use of CRFs to post-process the prediction results effectively improves the segmentation accuracy by 1.03%. The main reason is the image resolution of the self-built dataset. The rate is low

the dataset is relatively fuzzy

the features are highly complicated

and the labeling is inaccurate. The distribution of data is difficult to learn through the fully convolutional neural network

and the accuracy of the prediction results is low. Therefore

fully connected CRFs can improve the segmentation results to a large extent. Experimental results verify the effectiveness of the proposed network model and post-processing method.

Conclusion

This study mainly investigates the semantic segmentation of remote sensing images. The research belongs to computer vision and pattern recognition. The purpose is to let a computer identify the category of each pixel in the remote sensing image

namely

remote sensing image interpretation. Remote sensing image interpretation is a basic problem in remote sensing.It is an important means to obtain remote sensing image information

and the ground object information obtained from it can provide an important reference for various tasks

such as change detection and disaster relief. Improving the segmentation accuracy of remote sensing images has always been a popular topic. This study proposes a new network structure based on fully convolutional neural network for the characteristics of remote sensing images. On this basis

a sliding step prediction method based on integrated learning is proposed and used. Fully connected conditions are adopted for the post-processing of images to optimize the segmentation results and achieve a high-precision semantic segmentation of remote sensing images.

关键词

Keywords

references

Badrinarayanan V, Kendall A and Cipolla R.2017.SegNet:a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481-2495[DOI:10.1109/TPAMI.2016.2644615]

Blaschke T, Hay G J, Kelly M, Lang S, Hofmann P, Addink E, Feitosa R Q, Meer F, Werff H, Coillie F and Tiede D.2014.Geographic object-based image analysis-towards a new paradigm. ISPRS Journalof Photogrammetry and Remote Sensing, 87:180-191[DOI:10.1016/j.isprsjprs.2013.09.014]

Chen L C, Papandreou G and Kokkinos I.2017a. DeepLab:semantic image segmentation with deep convolutional nets, atrousconvolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-848[DOI:10.1109/TPAMI.2017.2699184]

Chen L C, Papandreou G and Kokkinos I.2014.Semantic image segmentation with deep convolutional nets and fully connected CRFs[EB/OL].[2019-01-04] . https://arxiv.org/pdf/1412.7062.pdf https://arxiv.org/pdf/1412.7062.pdf

Chen L C, Zhu Y K, Papandreou G, Schroff F and Adam H. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 801-818[ DOI: 10.1007/978-3-030-01234-2_49 http://dx.doi.org/10.1007/978-3-030-01234-2_49 ]

Chen L C, Papandreou G and Schroff F and Adam H. 2017b. Rethinking atrous convolution for semantic image segmentation[EB/OL].[2019-01-04] . https://arxiv.org/pdf/1706.05587.pdf https://arxiv.org/pdf/1706.05587.pdf

Chollet F. 2017. Xception: deep learning with depthwise separable convolutions[EB/OL].[2019-01-04] . https://arxiv.org/pdf/1610.02357.pdf https://arxiv.org/pdf/1610.02357.pdf

Du B and Zhang W.2016.Research on object-oriented high resolutionremotesensing image classification technology. Western Resources, (5):135-138

杜斌, 张炜. 2016.基于面向对象的高分辨率遥感影像分类技术研究.西部资源, (5):135-138

Everingham M, Zisserman A, Williams C K I, Gool L V, Allan M, Bishop C M, Chapelle O, Dalal N, Deselaers T, DorkóG, Stefan Duffner, Eichhorn J, Farquhar J D R, Fritz M, Garcia C, Griffiths T, Jurie F, Keysers D, Koskela M, Laaksonen J, Larlus D, Leibe B, Meng H Y, Ney H, Schiele B, Schmid C, Seemann E, Shawe-Taylor J, Storkey A, Szedmak S, Triggs B, Ulusoy I, Viitaniemi V and Zhang J G. 2005. The 2005 PASCAL Visual Object Classes Challenge//Machine Learning Challenges Workshop. Berlin, Heidelberg: Springer, 117-176[ DOI: 10.1007/11736790_8 http://dx.doi.org/10.1007/11736790_8 ]

Feng D C, Chen G, Xiao K L, Du W Y and Wu X Y.2012.Remote sensing image classification based on minimum distance method. Journal of North China Institute of Aerospace Engineering, 22(3):1-2, 5

冯登超, 陈刚, 肖楷乐, 杜文雅, 吴新颖.2012.基于最小距离法的遥感图像分类.北华航天工业学院学报, 22(3):1-2, 5)[DOI:10.3969/j.issn.1673-7938.2012.03.001]

Huang G, Liu Z, Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks[EB/OL].[2019-01-04] . https://arxiv.org/pdf/1608.06993.pdf https://arxiv.org/pdf/1608.06993.pdf

Jian Z, Cui Z and Liu A. 2007. A k-means remoter sensing image classification methed based on AdaBoost. Journal of Beijing Electronic Science&Technology Institute, 4:27-32

Liu H Q and Huete A. 1995. A feedback based modification of the NDVI to minimize canopy background and atmospheric noise. IEEE Transactions on Geoscience and Remote Sensing, 33(2), 457-465[DOI:10.1109/TGRS.1995.8746027]

Krähenbühl P and Koltun V.2011. Efficient inference in fully connected crfs with gaussian edge potentials[EB/OL].[2019-01-04] . https://arxiv.org/pdf/1210.5644.pdf https://arxiv.org/pdf/1210.5644.pdf

Kettig R L and Landgrebe D A. 1976. Classification of multispectral image data by extraction and classification of homogeneous objects. IEEE Transactions on Geoscience Electronics, 14(1), 19-26[DOI:10.1109/TGE.1976.294460]

Liu D W, Han L and Han X Y. 2016. High spatial resolution remote sensing image classification based on deep learning. Acta Optica Sinica, 36(4):#0428001

刘大伟, 韩玲, 韩晓勇. 2016.基于深度学习的高分辨率遥感影像分类研究.光学学报, 36(4):#0428001

Long J, Shelhamer E and Darrell T. 2014. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis&Machine Intelligence, 39(4):640-651

Lobo A, Chic O and Casterad A. 1996. Classification of Mediterranean crops with multisensor data:per-pixel versus per-object statistics and image segmentation. International Journal of Remote Sensing, 17(12), 2385-2400[DOI:10.1080/01431169608948779]

McFeeters S K. 1996. The use of the normalized difference water index (NDWI) in the delineation of open water features. International Journal of Remote Sensing, 17(7), 1425-1432[DOI:10.1080/01431169608948714]

Rouse J W Jr, Haas R H, Schell J A and Deering D W. 1974. Monitoring vegetation systems in the Great Plains with ERTS. Washington: NASA

Xiao Z X and Chen S.2018.Review of image semantic segmentation. Software Guide, 17(8):6-8, 12

肖朝霞, 陈胜.2018.图像语义分割问题研究综述.软件导刊, 17(8):6-8, 12

Ronneberger O, Fischer P and Brox T. 2015. U-Net: Convolutional networks for biomedical image segmentation//Proceedings of 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241[ DOI: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]

Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X and Cottrell G. 2018. Understanding convolution for semantic segmentation//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, NV, USA: IEEE: 1451-1460[ DOI: 10.1109/WACV.2018.00163 http://dx.doi.org/10.1109/WACV.2018.00163 ]

Yu F and Koltun V. 2015. Multi-scale context aggregation by dilated convolutions.[EB/OL].[2016-04-30] . https://arxiv.org/pdf/1511.07122 https://arxiv.org/pdf/1511.07122

Zhao H, Shi J, Qi X, Wang X and Jia J. 2017. Pyramid scene parsing network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 2881-2890[ DOI: 10.1109/CVPR.2017.660 http://dx.doi.org/10.1109/CVPR.2017.660 ]

Zhao X, Li Y and Zhao Q.2015. Mahalanobis distance based on fuzzy clustering algorithm for image segmentation. Digital Signal Processing, 43, 8-16[DOI:10.1016/j.dsp.2015.04.009]

Alert me when the article has been cited

提交

Language-guided cross-spatiotemporal domain adaptation for remote sensing image semantic segmentation

Semantic-guided contrastive learning for SAR and optical image translation

Underwater image segmentation methods： a survey

Semantic correction-based image inpainting under semisupervised learning

Enhanced attention-based joint semantic instance segmentation network for point clouds