This paper addresses the problem of fine-grained visual categorization requiring domain-specific and expert knowledge. This task is challenging because of all the objects in a database belonging to the same basic level category

with very fine differences between classes. These subtle differences are easily overcome by image background information that is seldom discriminative and often serves as a distractor

which reduces recognition performance in fine-grained image categorization. Therefore

segmenting the background regions and localizing discriminative image parts are important precursors to fine-grained image categorization. To this end

a new segmentation algorithm based on top-down attention maps is proposed to detect discriminative objects and discount the influence of background. Once the objects are localized

they are described by convolutional neural network (CNN) features to predict the corresponding class. The proposed method first recognizes a dataset through the CNN model to obtain a ConvNet basis with GoogLeNet structure. The ConvNet basis is visualized to build reliable intuitions of the visual information contained in the trained CNN representations

and the visualization result shows that the saliency parts in the images correspond to the object regions while the activations of background regions are very small. Then

according to the learned ConvNet

we predict the top-1 class label of a given image and determine the spatial support of the predicted class among the image pixels. Spatial support is rearranged to produce a top-down attention map

which can effectively locate the informative image object regions. Next

given an image and its corresponding image-specific class attention map

we compute the object segmentation mask with the GraphCut segmentation algorithm. The high-quality foreground segmentations are then used to encode the image appearance into a highly discriminative visual representation by finetuning the ConvNet basis to learn a new segmentation ConvNet. Finally

the ConvNet basis and segmentation ConvNet are combined to conduct fine-grained image recognition. We also use the original images to finetune the segmentation ConvNet for improved accuracy. The proposed model was tested on two new benchmark datasets available for fine-grained image categorization. The two databases were Cars196 and Aircrafts100

which were designed for fine-grained image recognition with public annotations

including class labels

object bounding boxes

and part locations. Only the class label annotation was used in our evaluation

and the final average accuracy rates of Cars196 and Aircrafts100 databases were 86.74% and 84.70%

respectively. These results show that adding visual attention information is more accurate than the GoogLeNet model alone. A semantic segmentation strategy based on top-down attention map was proposed to improve the accuracy of fine-grained image categorization. Our method did not need any bounding box or part annotations

making it very robust and applicable to a variety of datasets. The experimental results show that the attention information was very useful for fine-grained image recognition. The proposed novel model proved capable of application to salient object detection

foreground segmentation

and fine-grained image categorization.