Objective

2

Gatys et al. successfully used convolutional neural networks (CNNs) to render a content image in different styles in a process referred to as neural style transfer (NST). Their work was the first time deep learning demonstrated its ability in the field of style transfer. In the past

most related problems in style transfer were manually modeled

which was a time consuming and laborious process. The goal of traditional NST is to learn the mapping between two different styles of paired images. Cycle-consistent adversarial networks (CycleGAN) is the first method to apply the generative adversarial network (GAN) to image style transfer. This method has a good performance on unpaired training data but does not work well when the test image is different from the training images. Thus

instance style transfer was developed to address this problem. Instance style transfer is built on image segmentation and should be applied only on the object of interest. The main challenge is the transition between the object and a non-stylized background. Most studies on instance style transfer have focused on the CNN. In this paper

some of these methods are extended to CycleGAN

and some steps are improved based on actual conditions. We propose a method to achieve instance style transfer by combining fully convolutional network (FCN) with CycleGAN. A dataset is used to verify that training data are not the reason CycleGAN cannot work well on instance style transfer.

Method

2

This study is divided into two parts:The first part is to improve the performance of CycleGAN to make it work efficiently in instance style transfer. The second part is to verify the conjecture in the reference. In the first part

the FCN is utilized to obtain the semantic segmentation of input image

$$\mathit{\boldsymbol{X}}$$

. FCN must be trained by a large amount of labeled data in advance so that the network can segment the object with high accuracy. The output of the FCN is label image

$$\mathit{\boldsymbol{Y}}$$

. Next

CycleGAN is utilized for style transfer. In this step

CycleGAN must be trained with the prepared data to obtain style transfer image

$$\mathit{\boldsymbol{Z}}$$

. Then

the output of the CycleGAN

$$\mathit{\boldsymbol{Z}}$$

is matched with the label image

$$\mathit{\boldsymbol{Y}}$$

. When referring to image matching

we make the pixel points outside the areas of interests in the FCN's label image

$$\mathit{\boldsymbol{Y}}$$

to be zero and make the Hadamard product

$$\mathit{\boldsymbol{R}}$$

=

$$\mathit{\boldsymbol{Y}}$$

$$ \circ $$

$$\mathit{\boldsymbol{Z}}$$

. In this way

the areas of interests can be separated from the style transfer image

$$\mathit{\boldsymbol{Z}}$$

and

$$\mathit{\boldsymbol{R}}$$

is used to replace the pixel in the same location of the original image

$$\mathit{\boldsymbol{X}}$$

. For the second part

we create training sets for people riding horses

people beside horses

people riding zebras

or people with zebras to verify the problem raised by CycleGAN's author. Then

this dataset is used to train CycleGAN and observe the result. Data augmentation needs to be performed because images are hard to find.

Result

2

The first experiment shows that the recognition ability of CycleGAN improves considerably when combined with the FCN. The proposed method can achieve instance style transfer of the image

while the rest of the area in the image has minimal influence. In the experiment

we define an index that calculates the number of pixels changed outside the target object to measure the performance of instance style transfer and show the improvement of this method. A smaller index corresponds to improved performance of the instance style transfer. In the numerical simulations

the value of our method is smaller than that of the style transfer that uses CycleGAN only

thereby showing that the proposed method is more efficient in instance style transfer. In the second experiment

using our dataset to train CycleGAN shows that the CycleGAN is still unable to achieve instance style transfer. The network is difficult to train and oscillation of loss function is violent because of the complex color and background of the new training dataset. However

the instance style transfer performance of using a new dataset to train is improved.

Conclusion

2

FCN can obtain the semantic segmentation of an image. CycleGAN combined with FCN can achieve instance style transfer and ensure that the background and other objects remain unchanged. We verify that CycleGAN cannot accurately achieve instance style transfer of a given target when the test image is different from the training images.