Cross-media retrieval with hierarchical recurrent attention network

Jinwei Qi; Yuxin Peng; Yuxin Yuan

doi:10.11834/jig.180259

Column of NCIG'2018 | Views : 0 下载量: 0 CSCD: 1

Export
Share
Collection
Album

Cross-media retrieval with hierarchical recurrent attention network
Vol. 23, Issue 11, Pages: 1751-1758(2018)
Received：16 April 2018，

Revised：2018-7-1，

Published：16 November 2018
DOI： 10.11834/jig.180259
稿件说明：

移动端阅览

DOI：

Jinwei Qi, Yuxin Peng, Yuxin Yuan. Cross-media retrieval with hierarchical recurrent attention network[J]. Journal of Image and Graphics, 2018, 23(11): 1751-1758. DOI： 10.11834/jig.180259.

摘要

目的

跨媒体检索旨在以任意媒体数据检索其他媒体的相关数据，实现图像、文本等不同媒体的语义互通和交叉检索。然而，"异构鸿沟"导致不同媒体数据的特征表示不一致，难以实现语义关联，使得跨媒体检索面临巨大挑战。而描述同一语义的不同媒体数据存在语义一致性，且数据内部蕴含着丰富的细粒度信息，为跨媒体关联学习提供了重要依据。现有方法仅仅考虑了不同媒体数据之间的成对关联，而忽略了数据内细粒度局部之间的上下文信息，无法充分挖掘跨媒体关联。针对上述问题，提出基于层级循环注意力网络的跨媒体检索方法。

方法

首先提出媒体内-媒体间两级循环神经网络，其中底层网络分别建模不同媒体内部的细粒度上下文信息，顶层网络通过共享参数的方式挖掘不同媒体之间的上下文关联关系。然后提出基于注意力的跨媒体联合损失函数，通过学习媒体间联合注意力来挖掘更加精确的细粒度跨媒体关联，同时利用语义类别信息增强关联学习过程中的语义辨识能力，从而提升跨媒体检索的准确率。

结果

在2个广泛使用的跨媒体数据集上，与10种现有方法进行实验对比，并采用平均准确率均值MAP作为评价指标。实验结果表明，本文方法在2个数据集上的MAP分别达到了0.469和0.575，超过了所有对比方法。

结论

本文提出的层级循环注意力网络模型通过挖掘图像和文本的细粒度信息，能够充分学习图像和文本之间精确跨媒体关联关系，有效地提高了跨媒体检索的准确率。

Abstract

Objective

Cross-media retrieval aims to retrieve the data of different media types by a query

which can provide flexible and useful retrieval experience with numerous user demands at present. However

a "heterogeneity gap" leads to inconsistent representations of different media types

thus resulting in a challenging construction of correlation and realizing cross-media retrieval between them. However

data from different media types naturally have a semantic consistency

and their patches contain abundant fine-grained information

which provides key clues for cross-media correlation learning. Existing methods mostly consider a pairwise correlation of various media types with the same semantics

but they ignore the context information among the fine-grained patches

which cannot fully capture the cross-media correlation. To address this problem

a cross-media hierarchical recurrent attention network (CHRAN) is proposed to fully consider the intra-and inter-media fine-grained context information.

Method

First

we propose to construct a hierarchical recurrent network to fully exploit the cross-media fine-grained context information. Specifically

the hierarchical recurrent network consists of two levels

which are implemented by a long short-term memory network. We extract features from the fine-grained patches of different media types and organize them into sequences

which are considered the inputs of the hierarchical network. The bottom level aims to model the intra-media fine-grained context information

whereas the top level adopts a weight-sharing constraint to fully exploit inter-media context correlation

which aims to share the knowledge learned from different media types. Thus

the hierarchical recurrent network can provide intra-and inter-media fine-grained hints for boosting cross-media correlation learning. Second

we propose an attention-based cross-media joint embedding loss to learn a cross-media correlation. We utilize an attention mechanism to allow the models to focus on the necessary fine-grained patches within various media types

thereby allowing the inter-media co-attention to be explored. Furthermore

we jointly consider the matched and mismatched cross-media pairs to preserve the relative similarity ranking information. We also adopt a semantic constraint to preserve the semantically discriminative capability during the correlation learning process. Therefore

a precise fine-grained cross-media correlation can be captured to improve retrieval accuracy.

Result

We conduct experiments on two widely-used cross-media datasets

including Wikipedia and Pascal Sentence datasets

which consider 10 state-of-the-art methods for comprehensive comparisons to verify the effectiveness of our proposed CHRAN approach. We perform a cross-media retrieval with two types of retrieval tasks

that is

retrieving text by image and retrieving the image by text

and then we adopt mean average precision (MAP) score as the evaluation metric. We also conduct baseline experiments to verify the contribution of a weight-sharing constraint and cross-media attention modeling. The experimental results show that our proposed approach achieves the optimal MAP scores of 0.469 and 0.575 on two datasets and outperforms the state-of-the-art methods.

Conclusion

The proposed approach can effectively learn a fine-grained cross-media correlation precisely. Compared with the existing methods that mainly model the pairwise correlation and ignore the fine-grained context information

our proposed hierarchical recurrent network can fully capture the intra-and inter-media fine-grained context information with a cross-media co-attention mechanism that can further promote the accuracy of cross-media retrieval.

关键词

Keywords

references

Peng Y X, Huang X, Zhao Y Z. An overview of cross-media retrieval:concepts, methodologies, benchmarks and challenges[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2017.[DOI:10.1109/TCSVT.2017.2705068]

Hotelling H. Relations between two sets of variates[J]. Biometrika, 1936, 28(3-4):321-377.[DOI:10.2307/2333955]

Rasiwasia N, Costa Pereira J, Coviello E, et al. A new approach to cross-modal multimedia retrieval[C]//Proceedings of the 18th ACM International Conference on Multimedia. Firenze, Italy: ACM, 2010: 251-260.[ DOI:10.1145/1873951.1873987 http://dx.doi.org/10.1145/1873951.1873987 ]

Zhai X H, Peng Y X, Xiao J G. Learning cross-media joint representation with sparse and semisupervised regularization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(6):965-978.[DOI:10.1109/TCSVT.2013.2276704]

Feng F X, Wang X J, Li R F. Cross-modal retrieval with correspondence autoencoder[C]//Proceedings of the 22nd ACM International Conference on Multimedia. Orlando, Florida, USA: ACM, 2014: 7-16.[ DOI:10.1145/2647868.2654902 http://dx.doi.org/10.1145/2647868.2654902 ]

Peng Y X, Qi J W, Huang X, et al. CCL:cross-modal correlation learning with multigrained fusion by hierarchical network[J]. IEEE Transactions on Multimedia, 2018, 20(2):405-420.[DOI:10.1109/TMM.2017.2742704]

Hardoon D R, Szedmak S, Shawe-Taylor J. Canonical correlation analysis:An overview with application to learning methods[J]. Neural Computation, 2004, 16(12):2639-2664.[DOI:10.1162/0899766042321814]

Gong Y C, Ke Q F, Isard M, et al. A multi-view embedding space for modeling internet images, tags, and their semantics[J]. International Journal of Computer Vision, 2014, 106(2):210-233.[DOI:10.1007/s11263-013-0658-4]

Li D G, Dimitrova N, Li M K, et al. Multimedia content processing through cross-modal association[C]//Proceedings of the 11th ACM International Conference on Multimedia. Berkeley, CA, USA: ACM, 2003: 604-611.[ DOI:10.1145/957013.957143 http://dx.doi.org/10.1145/957013.957143 ]

Peng Y X, Zhai X H, Zhao Y Z, et al. Semi-supervised cross-media feature learning with unified patch graph regularization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(3):583-596.[DOI:10.1109/TCSVT.2015.2400779]

Ngiam J, Khosla A, Kim M, et al. Multimodal deep learning[C]//Proceedings of the 28th International Conference on Machine Learning. Bellevue, WA: ICML, 2011: 689-696. http://campar.in.tum.de/Students/MaDeepEMR .

Andrew G, Arora R, Bilmes J, et al. Deep canonical correlation analysis[C]//Proceedings of the 30th International Conference on Machine Learning. Atlanta, GA, USA: ACM, 2013: 1247-1255. https://www.mendeley.com/research-papers/deep-canonical-correlation-analysis/ .

Peng Y X, Huang X, Qi J W. Cross-media shared representation by hierarchical learning with multiple deep networks[C]//Proceeding of 25th International Joint Conference on Artificial Intelligence. New York, NY, USA: Morgan Kaufmann, 2016: 3846-3853.

Wei Y C, Zhao Y, Lu C Y, et al. Cross-modal retrieval with CNN visual features:a new baseline[J]. IEEE Transactions on Cybernetics, 2017, 47(2):449-460.[DOI:10.1109/TCYB.2016.2519449]

Wang B K, Yang Y, Xu X, et al. Adversarial cross-modal retrieval[C]//Proceedings of ACM on Multimedia Conference. Mountain View, California, USA: ACM, 2017: 154-162.[ DOI:10.1145/3123266.3123326 http://dx.doi.org/10.1145/3123266.3123326 ]

Huang X, Peng Y X, Yuan M K. Cross-modal common representation learning by hybrid transfer network[C]//Proceeding of the 26th International Joint Conference on Artificial Intelligence. Melbourne, Australia: Morgan Kaufmann, 2017: 1893-1900.[ DOI:10.24963/ijcai.2017/263 http://dx.doi.org/10.24963/ijcai.2017/263 ]

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv: 1409.1556, 2014. https://arxiv.org/abs/1409.1556 .

Lin Y T, Pang Z Y, Wang D H, et al. Task-driven visual saliency and attention-based visual question answering[J]. arXiv preprint arXiv: 1702.06700, 2017. https://arxiv.org/abs/1702.06700 .

Kim Y. Convolutional neural networks for sentence classification[C]//Proceeding of Conference on Empirical Methods in Natural Language Processing. Doha, Qatar: Association for Computational Linguistics, 2014: 1746-1751.[ DOI:10.3115/v1/D14-1181 http://dx.doi.org/10.3115/v1/D14-1181 ]

Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780.[DOI:10.1162/neco.1997.9.8.1735]

Lu J S, Yang J W, Batra D, et al. Hierarchical question-image co-attention for visual question answering[C]//Proceeding of Advances in Neural Information Processing Systems. Barcelona, Spain: MIT Press 2016: 289-297. http://export.arxiv.org/abs/1606.00061 .

Rashtchian C, Young P, Hodosh M, et al. Collecting image annotations using Amazon's Mechanical Turk[C]//Proceeding of NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Los Angeles, California: ACM, 2010: 139-147. https://www.mendeley.com/research-papers/collecting-image-annotations-using-amazons-mechanical-turk/ .

Kang C C, Xiang S M, Liao S C, et al. Learning consistent feature representation for cross-modal multimedia retrieval[J]. IEEE Transactions on Multimedia, 2015, 17(3):370-381.[DOI:10.1109/TMM.2015.2390499]

Alert me when the article has been cited

提交

Lightweight spaceborne remote sensing object detection algorithm with multi-attention mechanism

Cross-modal feature fusion and detail-enhanced RGB-D salient object detection

Offline recognition of handwritten mathematical expressions： a survey

Two-stage vision transformer for fusing global and local features in distracted driving behavior recognition

Multi-scale attention encoder-decoder network for low-dose dental CBCT denoising