Objective

2

With the rapid development of multimedia technology

the scale of multimedia data has been growing rapidly. For example

people are used to describing the things they want to show with multimedia data such as texts

images

and videos. Obtaining the relevant results of one modality using another modality is a good objective. In this sense

how to effectively perform semantic correlation analysis and measure the similarity between the data has gradually become a hot research topic. As the representation of different modal data is heterogeneous

it poses a great challenge to the cross-modal retrieval task. Hashing-based methods have received great attention in cross-modal retrieval because of its fast retrieval speed and low storage consumption. To solve the problem of heterogeneity between different modalities of the data

most of the current supervised hashing algorithms directly map different modal data into the Hamming space. However

these methods have the following limitations: 1) The data from each modality have different feature representations

and the dimensions of their feature spaces vary greatly. Therefore

it is difficult for these methods to obtain a consistent hash code by directly mapping the data from different modalities into the same Hamming space. 2) Although label information has been considered for these hashing methods

the structural information of the original data is ignored

which could result in a less-representative hash code to encode the original structural information in each modality. To solve these issues

a novel hashing algorithm called structure-preserving hashing with coupled projections (SPHCP) is proposed in this paper for cross-modal retrieval.

Method

2

Considering the heterogeneity between the cross-modal data

this algorithm first projects the data from different modalities into their respective subspaces to reduce the modal difference. A local graph model is also designed in the subspace learning to maintain the structural consistency between the samples. Then

to build a semantic relationship between different modalities

the algorithm maps the subspace features to the Hamming space to obtain a consistent hash code. At the same time

the label constraint is exploited to improve the discriminant power of the obtained hash codes. Finally

the algorithm measures the similarity of different modal data in terms of the Hamming distance.

Result

2

We compared our model with several state-of-the-art methods on three public datasets

namely

Wikipedia

MIRFlickr

and Pascal Sentence. The mean average precision (mAP) is used as the quantitative evaluation metric. We first test our method on two benchmark datasets

Wikipedia and MIRFlickr. To evaluate the impact of hash-code length on the performance of the algorithm

this experiment set the hash code length to 16

32

64

and 128 bits. The experimental results show that for both the text-retrieving image task and image-retrieving text task

our proposed method outperforms the existing methods in each length setting. To further measure the performance of our proposed method on the dataset with deep features

we test the algorithm on the Pascal Sentence dataset. The experimental results show that our SPHCP algorithm can also achieve higher mAP on such dataset with deep features. In general

cross-modal retrieval methods based on deep networks can handle nonlinear features well

so their retrieval accuracy is supposed to be higher than that of traditional methods

but they need much more computational power. As a "shallow" method

the proposed SPHCP algorithm is competitive with deep methods in terms of mAP. Therefore

as an interesting direction

our framework can be used in conjunction with the deep learning method in the future

i.e.

using deep learning to extract the features of images and text offline

and using the SPHCP algorithm for fast retrieval. Furthermore

we analyze the parameter sensitivity of the proposed algorithm. As this algorithm has 7 parameters

a controlled variable method is used for evaluation. The experimental results show that the proposed algorithm is not sensitive to parameters

which means that the training process does not require much optimization time

making it suitable for practical application.

Conclusion

2

In this study

a novel method called SPHCP is proposed to solve the problems mentioned. First

aiming at the "modal gap" between cross-modal data

the scheme of coupled projections is applied to gradually reduce the modal difference of multimedia data. In this way

a more consistent hash code can be obtained. Second

considering the structural information and semantic discrimination of the original data

the algorithm introduces the graph model in subspace learning

which can maintain the intra-class and inter-class relationship of the samples. Finally

a label constraint is introduced to improve the discriminability of the hash code. The experiments on the benchmark datasets verify the effectiveness of the proposed algorithm. Specifically

compared with the second-best method

SPHCP achieves an improvement by 6% and 3% on Wikipedia for two retrieval tasks. On MIRFlickr

SPHCP achieves an improvement by 2% and 5%. On Pascal Sentence

the improvement is approximately 10% and 7%. However

the proposed method requires a large amount of computing power when dealing with large-scale data

because SPHCP introduces a graph model to maintain the structural information between the data. The calculation of the structural information between each sample leads to a larger computing complexity.In future research

we will introduce nonlinear feature mapping into our SPHCP framework to improve its scalability when dealing with nonlinear feature data. Furthermore

we can extend the SPHCP from a cross-modal retrieval algorithm to a multi-modal version.