Eye fixation prediction combining with multiple attention mechanism

Li Kong; Xuemin Hu; Ding Wang; Yanfang Liu; Yan Zhang; Long Chen

doi:10.11834/jig.210590

Image Analysis and Recognition | Views : 0 下载量: 212 CSCD: 0

PDF
Export
Share
Collection
Album

Eye fixation prediction combining with multiple attention mechanism
Vol. 27, Issue 12, Pages: 3503-3515(2022)
Received：03 August 2021，

Revised：2021-12-27，

Accepted：03 January 2022，

Published：16 December 2022
DOI： 10.11834/jig.210590
稿件说明：

移动端阅览

DOI：

Li Kong, Xuemin Hu, Ding Wang, Yanfang Liu, Yan Zhang, Long Chen. Eye fixation prediction combining with multiple attention mechanism[J]. Journal of Image and Graphics, 2022, 27(12): 3503-3515. DOI： 10.11834/jig.210590.

摘要

目的

经典的人眼注视点预测模型通常采用跳跃连接的方式融合高、低层次特征，容易导致不同层级之间特征的重要性难以权衡，且没有考虑人眼在观察图像时偏向中心区域的问题。对此，本文提出一种融合注意力机制的图像特征提取方法，并利用高斯学习模块对提取的特征进行优化，提高了人眼注视点预测的精度。

方法

提出一种新的基于多重注意力机制(multiple attention mechanism，MAM)的人眼注视点预测模型，综合利用3种不同的注意力机制，对添加空洞卷积的ResNet-50模型提取的特征信息分别在空间、通道和层级上进行加权。该网络主要由特征提取模块、多重注意力模块和高斯学习优化模块组成。其中，空洞卷积能够有效获取不同大小的感受野信息，保证特征图分辨率大小的不变性；多重注意力模块旨在自动优化获得的低层丰富的细节信息和高层的全局语义信息，并充分提取特征图通道和空间信息，防止过度依赖模型中的高层特征；高斯学习模块用来自动选择合适的高斯模糊核来模糊显著性图像，解决人眼观察图像时的中心偏置问题。

结果

在公开数据集SALICON(saliency in context)上的实验表明，提出的方法相较于同结构的SAM-Res(saliency attention modal)模型以及DINet(dilated inception network)模型在相对熵(Kullback-Leibler divergence，KLD)、sAUC(shuffled area under ROC curve)和信息增益(information gain，IG)评价标准上分别提高了33%、0.3%和6%；53%、0.5%和192%。

结论

实验结果表明，提出的人眼注视点预测模型能通过加权的方式分别提取空间、通道、层之间的特征

在多数人眼注视点预测指标上超过了主流模型。

Abstract

Objective

Human eye fixation recognition has been developing in images-related computer vision in recent years. The distinctive salient regions of an image are selected for capturing visual structure better. Recent saliency models are developed through salient object detection

object segmentation and image cropping. Traditional applications are focused on hand-crafted features based on low-level cues (e.g.

contrast

texture

color) for saliency prediction. However

these features are easily failed to simulate the complex activation of the human visual system

especially in complex scenarios. Existing eye fixation prediction models often use jump connections to fuse high-level and low-level features

which easily leads to the difficulty of weighing the importance of features between different levels

and the gazing problem are biased toward the center. Commonly

humans are inclined to look at the center of the image when there are no obvious salient regions. We develop layer attention mechanism that different weights are assigned to different layer features for selective layer features extraction

and the channel attention mechanism and spatial attention mechanism are integrated to selectively extract different channel and spatial features in convolutional features. In addition

we facilitate a method of Gaussian learning to solve the problem of the center priors and improve the prediction accuracy.

Method

Our eye fixation prediction model is based on multiple attention mechanism network (MAM-Net)

which uses three different attention mechanisms to weight the feature information of different layers

different channels

and different image pixels extracted by the ResNet-50 model with dilated convolution. Our network is mainly composed of the feature extraction module

the novel multiple attention mechanism (MAM) module

and the Gaussian learning optimization module. 1) A dilated convolution network is used to capture long-range information via extracting local and global feature maps

which can contain a lot of different receptive fields. 2) A MAM attention module is incorporated features from different contexts of layer

channels

and image pixels of feature maps and output an intermediate saliency map. 3) A Gaussian learning layer is used to select best kernel automatically to blur the intermediate saliency map and generate the final saliency map. Our MAM module aims to optimize the obtained low-level features automatically in the context of rich details and high-level global semantic information features

fully extract channel and spatial information

and prevent over-reliance on high-level features. The Gaussian learning module is used for the final optimization processing since human eyes tend to focus to the image center

which is inconsistent with the prediction results of common methods. The deficiency of setting Gaussian fuzzy parameters is avoided by human prior in our method.

Result

Experiments on the public dataset saliency in context(SALICON) show that our results has improved Kullback-Leibler divergence (KLD)

shuffled area under region of interest(ROC) curve (sAUC)

and information gain (IG) evaluation criteria by 33%

0.3%

and 6%; 53%

0.6%

and 192%

respectively.

Conclusion

We propose a novel attentive model for predicting human eye fixations on natural images. Our MAM-Net can be used to predict saliency map of an image

which extract high-level and low-level features. The channel and spatial attention mechanism can optimize the feature maps of different layers

and the layer attention mechanism can predict the saliency map of the image composed of high-level and low-level features as well. We illustrate a Gaussian learning blur layer in terms of the integrated saliency maps optimization with different kernel.

关键词

Keywords

references

Borji A. 2021. Saliency prediction in the deep learning era: successes and Limitations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2): 679-700[DOI: 10.1109/TPAMI.2019.2935715]

Borji A and Itti L. 2012. Exploiting local and global patch rarities for saliency detection//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA: IEEE: 478-485[DOI: 10.1109/CVPR.2012.6247711 http://dx.doi.org/10.1109/CVPR.2012.6247711 ]

Bylinskii Z, Judd T, Oliva A, Torralba A and Durand F. 2019. What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(3): 740-757[DOI: 10.1109/TPAMI.2018.2815601 http://dx.doi.org/10.1109/TPAMI.2018.2815601 ]

Che Z H, Borji A, Zhai G T, Min X K, Guo G D and Le Callet P. 2020. How is gaze influenced by image transformations? Dataset and model. IEEE Transactions on Image Processing, 29: 2287-2300[DOI: 10.1109/TIP.2019.2945857]

Cheng M M, Mitra N J, Huang X L, Torr P H S and Hu S M. 2015. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3): 569-582[DOI: 10.1109/TPAMI.2014.2345401]

Cornia M, Baraldi L, Serra G and Cucchiara R. 2018. Predicting human eye fixations via an LSTM-based saliency attentive model. IEEE Transactions on Image Processing, 27(10): 5142-5154[DOI: 10.1109/TIP.2018.2851672]

Dorta G, Vicente S, Agapito L, Campbell N D F and Simpson I. 2018. Structured uncertainty prediction networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 5477-5485[DOI: 10.1109/CVPR.2018.00574 http://dx.doi.org/10.1109/CVPR.2018.00574 ]

Harel J, Koch C and Perona P. 2007. Graph-based visual saliency//Proceedings of the 19th International Conference on Neural Information Processing Systems. Vancouver, Canada: ACM: 545-552

He W and Pan C. 2022. The salient object detection based on attention-guided network. Journal of Image and Graphics, 27(4): 1176-1190

何伟, 潘晨. 2022. 注意力引导网络的显著性目标检测. 中国图象图形学报, 27(4): 1176-1190)[DOI: 10.11834/jig.200658]

Hu J, Shen L, Albanie S, Sun G and Wu E H. 2020. Squeeze-and-Excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8): 2011-2023[DOI: 10.1109/TPAMI.2019.2913372]

Huang X, Shen C Y, Boix X and Zhao Q. 2015. SALICON: reducing the semantic gap in saliency prediction by adapting deep neural networks//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE: 267-270[DOI: 10.1109/ICCV.2015.38 http://dx.doi.org/10.1109/ICCV.2015.38 ]

Judd T, Ehinger K, Durand F and Torralba A. 2009. Learning to predict where humans look//Proceedings of the 12th IEEE International Conference on Computer Vision. Kyoto, Japan: IEEE: 2106-2113[DOI: 10.1109/ICCV.2009.5459462 http://dx.doi.org/10.1109/ICCV.2009.5459462 ]

Kruthiventi S S S, Ayush K and Babu R V. 2017. DeepFix: a fully convolutional neural network for predicting human eye fixations. IEEE Transactions on Image Processing, 26(9): 4446-4456[DOI: 10.1109/TIP.2017.2710620]

Kümmerer M, Theis L and Bethge M. 2014. Deep gaze I: boosting saliency prediction with feature maps trained on ImageNet[EB/OL]. [2021-05-09] . https://arxiv.org/pdf/1411.1045.pdf https://arxiv.org/pdf/1411.1045.pdf

Liang M and Hu X L. 2015. Predicting eye fixations with higher-level visual features. IEEE Transactions on Image Processing, 24(3): 1178-1189[DOI: 10.1109/TIP.2015.2395713]

Liu N and Han J W. 2018. A deep spatial contextual long-term recurrent convolutional network for saliency detection. IEEE Transactions on Image Processing, 27(7): 3264-3274[DOI: 10.1109/TIP.2018.2817047]

Mahdi A and Qin J. 2019. An extensive evaluation of deep features of convolutional neural networks for saliency prediction of human visual attention. Journal of Visual Communication and Image Representation, 65: #102662[DOI: 10.1016/j.jvcir.2019.102662]

Oyama T and Yamanaka T. 2018. Influence of image classification accuracy on saliency map estimation. CAAI Transactions on Intelligence Technology, 3(3): 140-152[DOI: 10.1049/trit.2018.1012]

Tatler B W. 2007. The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. Journal of Vision, 7(14): #4[DOI: 10.1167/7.14.4]

Valenti R, Sebe N and Gevers T. 2009. Image saliency by isocentric curvedness and color//Proceedings of the 12th IEEE International Conference on Computer Vision. Kyoto, Japan: IEEE: 2185-2192[DOI: 10.1109/ICCV.2009.5459240 http://dx.doi.org/10.1109/ICCV.2009.5459240 ]

Vig E, Dorr M and Cox D. 2014. Large-Scale optimization of hierarchical features for saliency prediction in natural images//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 2798-2805[DOI: 10.1109/CVPR.2014.358 http://dx.doi.org/10.1109/CVPR.2014.358 ]

Wang W G, Shen J B and Jia Y D. 2019. Review of visual attention detection. Journal of Software, 30(2): 416-439

王文冠, 沈建冰, 贾云得. 2019. 视觉注意力检测综述. 软件学报, 30(2): 416-439)[DOI: 10.13328/j.cnki.jos.005636]

Woo S, Park J, Lee J Y and Kweon I S. 2018. CBAM: convolutional block attention module//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 3-19[DOI: 10.1007/978-3-030-01234-2_1 http://dx.doi.org/10.1007/978-3-030-01234-2_1 ]

Yang S, Lin G S, Jiang Q P and Lin W S. 2020. A dilated inception network for visual saliency prediction. IEEE Transactions on Multimedia, 22(8): 2163-2176[DOI: 10.1109/TMM.2019.2947352]

Zhang J M and Sclaroff S. 2016. Exploiting surroundedness for saliency detection: a Boolean map approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5): 889-902[DOI: 10.1109/TPAMI.2015.2473844]

Alert me when the article has been cited

提交

The salient object detection based on attention-guided network

Image colorization via color query and feature enhancement

Multi-granularity context network for SAR ship detection

Self-knowledge distillation for fine-grained image classification

High-security image steganography with the combination of multiple competition and channel attention