Object category prior knowledge is typically used to extract image features in popular approaches for the image description task based on the deep convolutional neural network (CNN) and long-short-term memory (LSTM) models.The task of image description involves detecting numerous objects in an image and generating a highly accurate description sentence.In general

people are always sensitive to objects in an image and direct considerable attention toward objects when they describe an image.However

the scene is also important in image description because objects are typically described within a specific scene (e.g.

place

location

or space)

or a sentence may lack semantic information

thereby leading to poor-quality description for a candidate image.In current popular approaches

the scene category and information are not regarded seriously

and scene category prior knowledge is always disregarded in an image.Consequently

the generated sentences do not provide a correct description for a scene and position relationships are easy to misjudge

which lead to the poor quality of these sentences.The effects of object category and scene category on generating a description sentence for an image is surveyed and studied to address the aforementioned problem.Both factors are determined to be useful for an accurate and semantic sentence.A novel framework

called fusion of scene and object category prior knowledge

in which scene category prior knowledge and object category prior knowledge are fused

is proposed and designed in this work based on the aforementioned findings to generate an accurate description of an image

improve the quality and semantics of a sentence

and achieve efficient performances. The objects and scene in an image are detected using CNN models

which are optimized on large-scale datasets

and the transfer learning method is mainly utilized for object category and scene category prior knowledge.A deep CNN model for scene recognition (CNN-Scene

abbreviated as CNN-S) is trained on the large-scale scene dataset Place205 to enable the CNN feature to include considerable scene category prior information.The parameters in CNN-S are transferred to CNN-S to extract the CNN feature for capturing scene category and information in a candidate image

and the parameters are continuously optimized using a fine-tuning method.Another deep CNN model (CNN-Object

abbreviated as CNN-O) is optimized on Imagenet

which is a large-scale dataset for object recognition.The parameters are applied to the CNN-O model through transfer learning and continuously trained via fine-tuning to determine object category and information in the image.The CNN features from CNN-S and CNN-O are inputted into the language models based on scene and based on object

respectively.These language models include two stacked LSTM layers and are constructed via factoring

in which the first layer receives the embedding features of words

and the outputs of the first layer and the CNN feature of the candidate image are combined to form multimodal features that will be inputted into the second LSTM layer.A fully connected layer is used as a classification layer to map the feature vector as category information.A Softmax function is used to compute the probability for each word in the vocabulary

which contains all the words that appear in the reference sentences of the image in the training dataset.A late fusion strategy by weighted average is applied to calculate the final score for each word

and the output on the current time step that corresponds to the maximum score is the expected word.All the words generated on all the time steps are formed into the description sentence for the candidate image. Three public popular datasets

namely

MSCOCO

Flickr30k

and Flickr8k

are used to evaluate the effectiveness of the proposed method.The using protocols of the three datasets and other popular approaches are followed.Three evaluation metrics

namely

BLEU for evaluating consistency and precision

METEOR for reflecting the precision and recall of words

and CIDEr for evaluating the semantic of candidate sentences

are adopted to evaluate the proposed model.Results on the three datasets demonstrate that the proposed model significantly improves performances on the three metrics.In particular

the performance on the CIDEr metric is 1.9% and 6.2% higher than that of object-and scene-based models

respectively

on the MSCOCO dataset.Performance is also improved on the Flickr30k dataset by 1.7% and 2.6% compared with the two benchmark models.For the Flickr8k dataset

CIDEr reaches 52.8%

thereby outperforming the two benchmark models by 9% and approximately 11%.On the BLEU and METEOR metrics

the proposed model also achieves better performances than the models with only object and scene as bases.The performances of the proposed model on the three datasets surpass those of most current state-of-the-art methods on multiple metrics. The experimental results show that the proposed model is effective.This model considerably improves performances compared with the benchmark models (object-based and scene-based models) and most popular state-of-the-art approaches.The results also indicate that the proposed model can generate numerous semantic description sentences for an image because the model captures the relationships between objects and scene when the scene category information is applied.However

the performance of the proposed model on the BLEU metric should be improved further.In a future work

additional prior knowledge

such as the action category and relationships among objects

will be fused into the proposed framework to further improve the quality of generation sentences for images

particularly the accuracy of sentences.Other novel techniques

such as residual net for effective CNN feature

region-based CNN for accurate object recognition

and gated recurrent unit for a concise language model

will be used to further improve performance.