浏览全部资源
扫码关注微信
收稿：2019-07-09，
修回：2019-9-16，
录用：2019-9-23，
纸质出版：2020-05-16
移动端阅览
目的
2
高分辨率遥感图像通常包含复杂的语义信息与易混淆的目标，对其语义分割是一项重要且具有挑战性的任务。基于DeepLab V3+网络结构，结合树形神经网络结构模块，设计出一种针对高分辨率遥感图像的语义分割网络。
方法
2
提出的网络结构不仅对DeepLab V3+做出了修改，使其适用于多尺度、多模态的数据，而且在其后添加连接树形神经网络结构模块。树形结构通过建立混淆矩阵、提取混淆图、构建图分割，能够对易混淆的像素更好地区分，得到更准确的分割结果。
结果
2
在国际摄影测量及遥感探测学会（International Society for Photogrammetry and Remote Sensing，ISPRS）提供的两个不同城市的遥感影像集上分别进行了实验，模型在整体准确率（overall accuracy，OA）这一项表现最好，在Vaihingen和Potsdam数据集上分别达到了90.4%和90.7%，其整体分割准确率较其基准结果有10.3%和17.4%的提升，对比ISPRS官方网站上的3种先进方法也有显著提升。
结论
2
提出结合DeepLab V3+和树形结构的卷积神经网络，有效提升了高分辨率遥感图像语义分割整体精度，其中易混淆类别数据的分割准确率显著提高。在包含复杂语义信息的高分辨率遥感图像中，由于易混淆类别之间的像素分割错误减少，使用了树形结构的网络模型的整体分割准确率也有较大提升。
Objective
2
High-resolution remote sensing image segmentation refers to the task of assigning a semantic label to each pixel in an image. Recently
with the rapid development of remote sensing technology
we have been able to easily obtain very-high resolution remote sensing images with a ground sampling distance of 5 cm to 10 cm. However
the very heterogeneous appearance of objects
such as buildings
streets
trees
and cars
in very-high-resolution data makes this task challenging
leading to high intraclass variance while the inter-class variance is low. A research hotspot is on detailed 2D semantic segmentation that assigns labels to multiple object categories. Traditional image processing methods depend on the extraction technique of the vectorization model
for example
based on region segmentation
line analysis
and shadow analysis. Another mainstream study relies on supervised classifiers with manually designed features. These models were not generalized when dealing with high-resolution remote sensing images. Recently
deep learning-based technology has helped explore the high-level semantic information in imaged and provide an end-to-end approach for semantic segmentation.
Method
2
Based on DeepLab V3+
we proposed an adaptive constructed neural network
which contains two connected modules
namely
the segmentation module and the tree module. When segmenting remote-sensing images
which contain multiscale objects
understanding the context is important. To handle the problem of segmenting objects at multiple scales
DeepLab V3+ employs atrous convolution in cascade or in parallel captures multiscale context by adopting multiple atrous rates. We adopted a similar idea in designing the segmentation module. This module uses an encoder-decoder architecture. The encoder is composed of four structures: EntryFlow
MiddleFlow
ExitFlow
and atrous spatial pyramid pooling(ASPP). In addition
the decoder is composed of two layers of SeparableConv blocks. The middle flow has two Xception blocks
which are linear stacks of depth-separable convolutional layers with residual connections. The segmentation module could capture well the multiscale features in the context. However
these features pay less attention to the easily confused classes. The other core contribution of the proposed method is the tree module. This module is constructed adaptively during the training. In each round
the method computes the confusion matrix on the evaluation data and calculates the confusion degrees between every two classes. A graph could be constructed according to the confusion matrix
and we can obtain a certain tree structure through the minimum cut algorithm. According to the tree structure
we build the tree module
in which each node is a ResNeXt unit. These nodes are connected by the concatenated connections. The tree module helped distinguish the pixels between easily confused classes by adding several neural layers to process their features. To implement the proposed method
the segmentation model is based on the MXNet framework and uses two Nvidia GeForce GTX1080 Ti graphic cards for accelerated training. The input size of the image block is 640×640 pixels due to memory limitation. We set the momentum (momentum) to 0.9
the initial learning rate to 0.01
adjust the learning rate to 0.001 when the training reaches half
and adjust the learning rate to 0.000 1 when the training reaches 3/4. We perform data augmentation before training due to the small amount of data in the ISPRS(International Society for Photogrammetry and Remote Sensing) remote sensing dataset. For each piece of raw data
we rotate the image center by 10° each time and cut out the largest square tile. In this way
each training image can obtain 36 sets of pictures after rotation. In addition
because the original training image is very large in size
to directly place the entire image into the network for training is not possible. Thus
it needs to be cropped into image blocks of 640×640 pixels. We apply an overlap-tile strategy to ensure no obvious cracks in the segmentation result map after splicing.
Result
2
The model
in this study performed the best in terms of overall accuracy (OA)
which reached 90.4% and 90.7% on the Vaihingen and Potsdam datasets
respectively. This result indicates that the model can achieve high segmentation accuracy. In addition
in the performance of easily confused categories
for example
low shrub vegetation (low_veg) and trees
F
1
values have greatly improved. In experiments of the Vaihingen dataset
the F
1
values of the low shrub vegetation and tree species reached 83.6% and 89.6%
respectively. In experiments of the Potsdam dataset
the F
1
values of the low shrub vegetation and tree species reached 86.8% and 87.1%. As for the average F
1
value
the scores of the model on the Vaihingen and Potsdam datasets are 89.3% and 92.0%
respectively. This number is much higher than the other latest methods
indicating that the model in this study is the best for both the segmentation of remote sensing images and the average performance of each category. Additionally
compared with the model without the tree module
the proposed method has higher segmentation accuracy for each category. The OA by using tree module increased by 1.1%
and the average F
1
value increased by 0.6% in the Vaihingen dataset. The OA and the average F
1
value increased by 1.3% and 0.9% in the Potsdam dataset. The result shows that the tree module does not only target a certain category but also improves the overall segmentation accuracy.
Conclusion
2
The proposed network can effectively improve the overall semantic segmentation accuracy of high-resolution remote sensing images. The experimental results show that the proposed segmentation module with the tree module is greatly improved due to the reduction of the error on easily confused pixels. The proposed method in this study is universal and suitable for a wide range of application scenarios.
Audebert N, Le Saux B and Lefèvre S. 2018. Beyond RGB:very high resolution urban remote sensing with multimodal deep networks. ISPRS Journal of Photogrammetry and Remote Sensing, 140:20-32[DOI:10.1016/j.isprsjprs.2017.11.011]
Badrinarayanan V, Kendall A and Cipolla R. 2017. SegNet:a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481-2495[DOI:10.1109/TPAMI.2016.2644615]
Boykov Y and Kolmogorov V. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(9):1124-1137[DOI:10.1109/TPAMI.2004.60]
Chen L C, Papandreou G, Schroff F and Adam H. 2017. Rethinking Atrous Convolution for Semantic Image Segmentation[EB/OL ] .[2019-07-01 ] . https://arxiv.org/pdf/1706.05587.pdf https://arxiv.org/pdf/1706.05587.pdf
Chen L C, Zhu Y K, Papandreou G, Schroff F and Adam H. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 833-851[ DOI:10.1007/978-3-030-01234-2_49 http://dx.doi.org/10.1007/978-3-030-01234-2_49 ]
Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S and Schiele. 2016. The cityscapes dataset for semantic urban scene understanding//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 3213-3223[ DOI:10.1109/CVPR.2016.350 http://dx.doi.org/10.1109/CVPR.2016.350 ]
Gerke M. 2014. Use of the Stair Vision Library within the ISPRS 2D Semantic Labeling Benchmark (Vaihingen). Holland: University of Twente
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 770-778[ DOI:10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Hu W, Huang Y Y, Wei L, Zhang F and Li H C. 2015. Deep convolutional neural networks for hyperspectral image classification. Journal of Sensors, 2015:#258619[DOI:10.1155/2015/258619]
Kampffmeyer M, Salberg A B and Jenssen R. 2016. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolut ional neural networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Las Vegas, NV, USA:IEEE: 680-688[ DOI:10.1109/CVPRW.2016.90 http://dx.doi.org/10.1109/CVPRW.2016.90 ]
Krizhevsky A, Sutskever I and Hinton G E. 2012. ImageNet classification with deep convolutional neural networks//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada, USA Curran Associates Inc: 1097-1105
Li R R, Liu W J, Yang L, Sun S H, Hu W, Zhang F and Li W. 2018. Deepunet:a deep fully convolutional network for pixel-level sea-land segmentation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(11):3954-3962[DOI:10.1109/JSTARS.2018.2833382]
Li X D, Ye M and Li T. 2017. Review of object detection based on convolutional neural networks. Application Research of Computers, 34(10):2881-2886, 2891
李旭冬, 叶茂, 李涛. 2017.基于卷积神经网络的目标检测研究综述.计算机应用研究, 34(10):2881-2886, 2891[DOI:10.3969/j.issn.1001-3695.2017.10.001]
Long J, Shelhamer E and Darrell T. 2015. Fully convolutional networks for semantic segmentation//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE: 3431-3440[ DOI:10.1109/CVPR.2015.7298965 http://dx.doi.org/10.1109/CVPR.2015.7298965 ]
Maggiori E, Tarabalka Y, Charpiat G and Alliez. 2016. Fully convolutional neural networks for remote sensing image classification//Proceedings of 2016 IEEE International Geoscience and Remote Sensing Symposium. Beijing, China: IEEE: 5071-5074[ DOI:10.1109/IGARSS.2016.7730322 http://dx.doi.org/10.1109/IGARSS.2016.7730322 ]
Noh H, Hong S and Han B. 2015. Learning deconvolution network for semantic segmentation//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 1520-1528[ DOI:10.1109/ICCV.2015.178 http://dx.doi.org/10.1109/ICCV.2015.178 ]
Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical image computing and computer-assisted intervention. Munich, Germany: Springer: 234-241[ DOI:10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]
Sherrah J. 2016. Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery[EB/OL ] .[2019-07-01 ] . https://arxiv.org/pdf/1606.02585.pdf https://arxiv.org/pdf/1606.02585.pdf
Sun J Y, Huang Z J, Zhou S G, Xu N, Qian H M and Wang C L. 2017. Building outline vectorization from high spatial resolution imagery. Journal of Remote Sensing, 21(3):396-405
孙金彦, 黄祚继, 周绍光, 徐南, 钱海明, 王春林. 2017.高分辨率遥感影像中建筑物轮廓信息矢量化.遥感学报, 21(3):396-405[DOI:10.11834/jrs.20176127]
Tan K, Wang X and Du P J.2019. Research progress of the remote sensing classification combining deep learning and semi-supervised learning. Journal of Image and Graphics, 24(11):1823-1841
谭琨, 王雪, 杜培军. 2019.结合深度学习和半监督学习的遥感影像分类进展.中国图象图形学报, 24(11), 1823-1841[DOI:10.11834/jig.190348]
Volpi M and Tuia D. 2017. Dense semantic labeling of subdecimeter resolution images with convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing, 55(2):881-893[DOI:10.1109/TGRS.2016.2616585]
Wang H, Tong H J, Zuo B X and Tang W R. 2014. Integration of multiscale segmentation algorithm and vectorization algorithm for remote sensing image. Computer Engineering, 40(6):256-261
王海, 童恒建, 左博新, 汤文瑞. 2014.遥感图像多尺度分割算法与矢量化算法的集成.计算机工程, 40(6):256-261[DOI:10.3969/j.issn.1000-3428.2014.06.055]
Wang L, Zhang F, Li W, Xie X M and Hu W. 2015. A method of SAR target recognition based on Gabor filter and local texture feature extraction. Journal of Radars, 4(6):658-665
王璐, 张帆, 李伟, 谢晓明, 胡伟. 2015.基于Gabor滤波器和局部纹理特征提取的SAR目标识别算法.雷达学报, 4(6):658-665[DOI:10.12000/JR15076]
Wu J, Cheng M M, Yao Z X, Peng Z Y, Li J and Ma J. 2015. Automatic generation of high-quality urban DSM with airborne oblique images. Journal of Image and Graphics, 20(06):117-128
吴军, 程门门, 姚泽鑫, 彭智勇, 李俊, 马峻. 2015.倾斜航空影像的城区DSM生成.中国图象图形学报, 20(06), 117-128[DOI:10.11834/jig.20150615]
Xie S N, Girshick R, Dollár P, Tu Z W and He K M. 2017. Aggregated residual transformations for deep neural networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 5987-5995[ DOI:10.1109/CVPR.2017.634 http://dx.doi.org/10.1109/CVPR.2017.634 ]
Zhang K, Hei B Q, Zhou Z and Li S Y. 2018. CNN with coefficient of variation-based dimensionality reduction for hyperspectral remote sensing images classification. Journal of Remote Sensing, 22(1):87-96
张康, 黑保琴, 周壮, 李盛阳. 2018.变异系数降维的CNN高光谱遥感图像分类.遥感学报, 22(1):87-96[DOI:10.11834/jrs.20187075]
相关作者
相关机构
京公网安备11010802024621
