In order to overcome the semantic gap between low-level features and high-level semantic concepts of imagery

a new image annotation refinement approach based on Random Dot Product Graph (RDPG)is proposed. In our approach

the visual features of images are used to construct a semantic graph of the candidate annotations. Then

we reconstruct the semantic graph with a RDPG

find the unobserved relevance in the incompletely observed semantic graph

and transform the random graph into the probabilities of state transition. Combined with Random Walk with Restart (RWR)

the final annotations are chosen. This new method incorporates the visual and semantic information of images

and reduces the influence of the scale of database. Experiments conducted on three standard databases demonstrate that our approach outperforms the existing image annotation refinement techniques. The macro F-Score and micro average F-Score can reach 0.784 and 0.743 respectively.