Objective

2

Semantic segmentation is a core computer vision task where one aims to densely assign labels to each pixel in the input image

such as person

car

road

pole

traffic light

or tree. Convolutional neural network-based approaches achieve state-of-the-art performance on various semantic segmentation tasks with applications for autonomous driving

image editing

and video monitoring. Despite such progress

these models often rely on massive amounts of pixel-level labels. However

for a real urban scene task

large amounts of labeled data are unavailable because of the high labor of annotating segmentation ground truth. When the labeled dataset is difficult to obtain

adversarial-training-based methods are preferred. These methods seek to adapt by confusing the domain discriminator with domain alignment standalone from task-specific learning under a separate loss. Another challenge is that a large difference exists between source data and target data in real scenarios. For instance

the distribution of appearance for objects and scenes may vary in different places

and even weather and lighting conditions can change significantly at the same place. In particular

such differences are often called as "domain gaps" and could cause significantly decreased performance. Unsupervised domain adaptation seeks to overcome such problems without target domain labels. Domain adaption aims to bridge the source and target domains by learning domain-invariant feature representations without using target labels. Such efforts have been made by using a deep learning network like AdaptSegNet

which has gained good results in the semantic segmentation of urban scenes. However

the network is trained directly using the synthetic dataset GTA5 and the real urban scene dataset Cityscapes

which exhibit a domain gap in gray

structure

and edge information. The fixed learning rate is employed in the model during the adversarial learning of different feature layers. In sum

segmentation accuracy needs to be improved.

Method

2

To handle these problems

a new domain adaptation method is proposed for urban scene semantic segmentation. To reduce the domain gap between source and target datasets

knowledge transfer or domain adaption is proposed to close the gap between source and target domains. This work is based on adversarial learning. First

the semantic-aware grad-generative adversarial network(GAN) (SG-GAN) is introduced to pre-process the synthetic dataset of GTA5. As a result

a new dataset SG-GTA5 is generated

which brings the newly dataset SG-GTA5 considerably closer to the urban scene dataset Cityscapes in gray

structure

and edge information. It is also suitable to substitute the original dataset GTA5 in AdaptSegNet. Second

the newly dataset SG-GTA5 is used as input of our network. To further enhance the adapted model and handle the fixed learning rate of AdaptSegNet

a multi-level adversarial network is constructed to effectively perform output space domain adaptation at different feature levels. Third

an adaptive learning rate is introduced in different feature levels of the network. Fourth

the loss value of different levels is adjusted by the proposed adaptive learning rate. Thus

the network's parameters can be updated dynamically. Fifth

a new convolution layer is added into the discriminator of GAN. As a result

the discriminant ability of the network is enhanced. For the discriminator

we use an architecture that uses fully convolutional layers to replace all fully connected layers to retain the spatial information. This architecture is composed of six convolution layers with 4×4 kernel and a stride of 2 for the first four kernels

a stride of 1 for the fifth kernel and channel numbers of 64

128

256

512

1 024

and 1

respectively. Except for the last layer

each convolution layer is followed by a leaky ReLU parameterized by 0.2. An up-sampling layer is added to the last convolution layer for re-scaling the output to the input size. No batch-normalization layers are used because we jointly train the discriminator with the segmentation network using a small batch size. For the segmentation network

it is essential to build upon a good baseline model to achieve high-quality segmentation results. We adopt the DeepLab-v2 framework with ResNet-101 model pre-trained on ImageNet as our segmentation baseline network. Similar to the recent work on semantic segmentation

we remove the last classification layer and modify the stride of the last two convolution layers from 2 to 1

making the resolution of the output feature maps effectively 1/8 times the input image size. To enlarge the receptive field

we apply dilated convolution layers in conv4 and conv5 layers with a stride of 2 and 4

respectively. After the last layer

we use the atrous spatial pyramid pooling (ASPP)as the final classifier. Finally

the batch normalization(BN) layers are removed because the discriminator network is trained with small batch generator network. Furthermore

we implement our network using the PyTorch toolbox on a single GTX1080Ti GPU with 11 GB memory.

Result

2

The new model is verified using the Cityscapes dataset. Experimental results demonstrate that the presented model is capable of segmenting more complex targets in the urban traffic scene precisely. The model also performs well against existing state-of-the-art segmentation model in terms of accuracy and visual quality. The segmentation accuracy of sidewalk

wall

pole

car and sky are improved by 9.6%

5.9%

4.9%

5.5%

and 4.8%

respectively.

Conclusion

2

The effectiveness of the proposed model is validated by using the real urban scene Cityscapes. The segmentation precision is improved through the presented dataset preprocessing scheme on the synthetic dataset of GTA5 by using the SG-GAN model

which makes the newly dataset SG-GTA5 much closer to the urban scene dataset Cityscapes on gray

structure

and edge information. The presented data preprocessing method also reduces the adversarial loss value effectively and avoids gradient explosion during the back propagation process. The network's learning capability is also further strengthened and the model's segmentation precision is improved through the presented adaptive learning rate

which is used for different adversarial layers to adjust the loss value of each layer. The learning rate can also update network parameters dynamically and optimize the performance of the generator and discriminator network. Finally

the discrimination capability of the proposed model is further improved by adding a new convolution layer in the discriminator

which enables the model to learn high layer semantic information. The domain shift is also alleviated to some extent.