Objective

2

Image description

a popular research field in computer vision and natural language processing

focuses on generating sentences that accurately describe the image content. Image description has a wide application in infant education

film production

and road navigation. In previous methods

generating image description based on deep learning achieved great success. On the basis of encoder-decoder framework

convolutional neural network is used as the feature extractor

and recurrent neural network is used as the caption generator. Cross entropy is applied to calculate the generation loss. However

the descriptions produced by these methods are often overly disorderly and inaccurate. Some researchers have exposed regularization method based on the encoder-decoder framework to strengthen the relationship between the image and generated description. However

incoherent problems remain with the generated descriptions caused by missing local information and high-level semantic concepts. Therefore

we propose a novel fusion training method based on encoder-decoder framework and generative adversarial networks

which enables global and local information to be calculated by a generator and inhibitor. This method encourages high linguistic coherence to human level while closing semantic concepts between image and description.

Method

2

The model is composed of an image feature extractor

inhibitor

generator

and discriminator. First

ResNet-152 is used as the image feature extractor. In ResNet-152

a key module named bottleneck is made up of a 1×1 convolution layer in 64 dimensions

a 3×3 convolution layer in 64 dimensions

a 1×1 convolution layer in 256 dimensions

and a shortcut connection. To suppress the time complexity per layer

an important principle is that the number of filters will be doubled if the dimension of convolutions is halved. The shortcut connection is introduced to address vanishing gradient. The last layer in ResNet-152 is replaced by a fully connected layer to align the dimension between the image feature and word after embedding. For the input of an image

the output of the extractor is an image feature vector with 512 dimensions. Second

the inhibitor is composed of a long short-term memory (LSTM). The input of the first moment is an image feature from the extractor. Every moment after that

the input is a word vector after embedding from ground truth. The results of inhibitor and "ground truth" are used to calculate the local score. The local score represents the coherence of the generated sentences. Third

the structure of the generator is the same as that of the inhibitor. Despite parameter sharing between the inhibitor and generator

the input of the generator is different from that of the inhibitor. In the generator

the output of the previous moment is used as the input of the current moment. The generator result is the image description and is used as part of the discriminator input. Fourth

the discriminator similarly consists of LSTM. Each word in the description generated by the generator corresponds to the input of the discriminator at each moment. The discriminator output at the last moment is combined with the image features obtained by the feature extractor to calculate the global score. The global score measures the semantic similarity between the generated description and image. Finally

the fusion loss consists of local and global scores. By controlling the weight of the local and global scores in fusion loss

coherence and accuracy are given different degrees of attention

and different descriptions may be generated for the same image. On the basis of the fusion loss

the model optimizes the parameter by backpropagation. In the experiment

the feature extractor is pretrained on the basis of the ImageNet dataset

and the parameters of the last layer are fine-tuned in formal training. As the training number increases

the generated sentences will perform increasingly well in terms of coherence and accuracy.

Result

2

Model performance is evaluated using three datasets

namely

MSCOCO-2014

Oxford-102

and CUB-200-2011. In the CUB-200-2011 dataset

our method shows improvement compared with that using the maximum likelihood estimate as the optimization function (CIDEr +1.6%

BLEU-3 +0.2%

BLEU-2 +0.8%

BLEU-1 +0.7%

and ROUGE-L +0.5%). The model performance declines when the inhibitor is removed from the model (BLEU-4 -0.8%

BLEU-3 -1.2%

BLEU-2 -1.6%

BLEU-1 -0.9%

ROUGE-L -1.8%

and METEOR -1.0%). In the Oxford-102 dataset

our method gains additional improvements compared with that using MLE as the optimization function (CIDEr +3.6%

BLEU-4 +0.7%

BLEU-3 +0.6%

BLEU-2 +0.4%

BLEU-1 +0.2%

ROUGE-L +0.6%

and METEOR +0.7%). The model performance declines substantially after removing the inhibitor (CIDEr -3.8%

BLEU-4 -1.5%

BLEU-3 -1.7%

BLEU-2 -1.4%

BLEU-1 -1.5%

ROUGE-L -0.5%

and METEOR -0.1%). In the MSCOCO-2014 dataset

our method achieves a leading position in several evaluation metrics compared with several proposed methods (BLEU-3 +0.4%

BLEU-2 +0.4%

BLEU-1 +0.4%

and ROUGE-L +0.3%).

Conclusion

2

The new optimization function and fusion training method are considered in the dependency relationships between words and the semantic relativity between the generated description and image. The method in this study obtains better scores in multiple evaluation metrics than that using MLE as the optimization function. Experiment results indicate that the inhibitor has a positive effect on the model performance in terms of evaluation metrics while optimizing the coherence of generated descriptions. The method increases the coherency of generated captions on the premise of the captions generated by the generator and image matching in the content.