Region enhancement and multi-feature fusion for contraband recognition in X-ray images

Ping Han; Hui Yang; Cheng Fang

doi:10.11834/jig.211134

Image Analysis and Recognition | Views : 0 下载量: 151 CSCD: 0

PDF
Export
Share
Collection
Album

Region enhancement and multi-feature fusion for contraband recognition in X-ray images
Vol. 28, Issue 2, Pages: 430-440(2023)
Received：06 December 2021，

Revised：2022-3-9，

Accepted：16 March 2022，

Published：16 February 2023
DOI： 10.11834/jig.211134
稿件说明：

移动端阅览

DOI：

Ping Han, Hui Yang, Cheng Fang. Region enhancement and multi-feature fusion for contraband recognition in X-ray images[J]. Journal of Image and Graphics, 2023, 28(2): 430-440. DOI： 10.11834/jig.211134.

摘要

目的

对旅客行李进行安全检查是维护公共安全的措施之一，安检智能化是未来的发展方向。基于X光图像的安检因不同的安检机成像方式不同，同一类违禁品在不同设备上的X光图像在颜色分布上有很大差异，导致安检图像智能识别算法在训练与测试数据分布不同时，识别性能明显降低，同时X光行李图像中物品的混乱复杂增加了违禁品识别的难度。针对上述问题，本文提出一种区域增强和多特征融合模型。

方法

首先，通过注意力机制的思想提取一种区域增强特征，消除颜色分布不同的影响，保留图像整体结构并增强违禁品区域信息。然后，采用多特征融合策略丰富特征信息，使模型适用于图像中物品混乱复杂情况。最后，提出一种三元损失函数优化特征融合。

结果

在公开数据集SIXray数据集上进行整体识别性能和泛化性能的实验分析，即测试本文方法在相同和不同颜色分布样本上的性能。在整体识别性能方面，本文方法在平均精度均值(mean average precision，mAP)上相较于基础模型ResNet18和ResNet34分别提升了4.09%和2.26%，并优于一些其他识别方法。对于单类违禁品，本文方法在枪支和钳子类违禁品上的平均识别精度为94.25%和90.89%，相较于对比方法有明显优势。在泛化性能方面，本文方法在SIXray_last101子数据集上可正确识别26张含违禁品样本，是基础模型能够正确识别数量的4.3倍，表明本文方法在颜色分布不同样本上的有效性。

结论

本文方法根据X光安检图像颜色差异的特点设计出一种区域增强特征，并与彩色和边缘特征融合，以获取多元化信息，在枪支、刀具、钳子等违禁品的识别任务中表现出较好效果，有效缓解了图像颜色分布差异导致的性能下降问题。

Abstract

Objective

X-ray security screening technology is widely used in public transportation infrastructures. The real-time security images are generated via X-ray-related scanning for checking. Due to manual inspection mechanism has its hidden risks

it is required to develop prohibited items-related intelligent recognition based on X-ray security check images.

Method

The convolutional neural network based (CNN-based) technique has been developing dramatically in the field of computer vision tasks. The CNN-based intelligent recognition model is restricted by a huge amount of label-manual X-ray images for training. Current recognition model is just suitable for homogeneous data-distributed between the training and testing sets. When the color distribution of X-ray images in the testing set is inconsistent in the training set

it is difficult to identify the target for the model. However

the problem of multiple datasets distribution is more prominent in practice for such application scenarios. Dual-energy X-ray imaging technique allows the scanner to distinguish different colors in terms of the item's effective atomic number. The heterogeneity problem of X-ray images is challenged in color distribution. The performance of X-ray image intelligent recognition algorithm will be lower intensively when the distribution of training and test data is inconsistent. So

we develop a heterogeneity-alleviated multi-feature fusion model further. First

to alleviate the influence of different color distribution of prohibited items

the attention mechanism is adopted to extract a newly pixel-level feature

called region-enhanced feature

which are trained in terms of overall feature distribution. The generalization ability is improved for multicolor-distributed X-ray images. Then

multi-feature fusion strategy is used to enrich the feature information like color

shape and outline. The features of color

edge and region-enhanced are melted into a centralized manner. The balanced weight parameters are added to the three kinds of features. Multi-feature fusion can be used to realize more effective feature information and optimal robustness in the case of chaotic objects in an image. Finally

a ternary loss functions are illustrated in relevant to fusion

edge and regional enhancement. To get feature fusion better

weight of three losses are set to balance the weighted feature-parameters.

Result

The experimental analysis is carried out on the public dataset for the performance evaluation of entirety and generalization (i.e.

performance on test samples with the same and different color distributions)

called SIXray. Our mean average precision (mAP) can be improved by 4.09% and 2.26% of each in comparison with the ResNet18 and ResNet34. For a single class prohibited items

the average accuracy of our method can reach 94.25% and 90.89% in the identification of guns and pliers-prohibited. We can identify 26 samples in SIXray_last101 dataset in generalization

which is 4.3 times beyond benchmark. The demonstration shows the effectiveness is improved in terms of multicolor-distributed samples. Additionally

ablation experiments are conducted to verify the effects of multiple features and hyper-parameter settings. The experimental results show that the overall recognition performance can be improved based on the richer multiple features (each of edge features and regional enhancement features improve the overall recognition performance by 1.32 and 1.05 percentage points).

Conclusion

A region-enhanced multi-feature fusion method is developed to deal with rich color and different distribution and chaotic and complex objects through X-ray security images-relevant feature analysis. The enhanced region features are obtained in terms of feature distribution overall. Multi-feature fusion strategy is implemented for the optimization of color

shape and contour details. And

a ternary loss function is used to improve the fusion effect and its heterogeneity. Our analyses demonstrate that the performance of the model can be improved for prohibited items checking. The effectiveness and robustness of the proposed method are verified as well. The multi-branch structure of the model is required to be developed further due to its limitations of computational cost and recognition efficiency.

关键词

Keywords

references

Akcay S and Breckon T. 2022. Towards automatic threat detection: a survey of advances of deep learning within X-Ray security imaging. Pattern Recognition, 122: #108245 [DOI: 10.1016/J.PATCOG.2021.108245]

Akçay S, Kundegorski M E, Devereux M and Breckon T P. 2016. Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery//Proceedings of 2016 IEEE International Conference on Image Processing (ICIP). Phoenix, USA: IEEE: 1057-1061 [ DOI: 10.1109/ICIP.2016.7532519 http://dx.doi.org/10.1109/ICIP.2016.7532519 ]

Baştan M, Yousefi M R and Breuel T M. 2011. Visual words on baggage X-ray images//Proceedings of the 14th International Conference on Computer Analysis of Images and Patterns. Seville, Spain: Springer: 360-368 [ DOI: 10.1007/978-3-642-23672-3_44 http://dx.doi.org/10.1007/978-3-642-23672-3_44 ]

Caldwell M, Ransley M, Rogers T W and Griffin L D. 2017. Transferring x-ray based automated threat detection between scanners with different energies and resolution//Proceedings Volume 10441, Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies. Warsaw, Poland: SPIE: #104410F [ DOI: 10.1117/12.2277641 http://dx.doi.org/10.1117/12.2277641 ]

Duta I C, Liu L, Zhu F and Shao L. 2020. Pyramidal convolution: rethinking convolutional neural networks for visual recognition [EB/OL]. [2021-08-28] . https://arxiv.org/pdf/2006.11538v1.pdf https://arxiv.org/pdf/2006.11538v1.pdf

Hassan T, Akçay S, Bennamoun M, Khan S and Werghi N. 2021. Unsupervised anomaly instance segmentation for baggage threat recognition. Journal of Ambient Intelligence and Humanized Computing, 1-12 [DOI: 10.1007/S12652-021-03383-7]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 770-778 [ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

He Y H, Wu G X and Wu Z Q. 2021. Domain alignment based object detection of X-ray images. Computer Science, 48(1): 175-181

何彦辉, 吴桂兴, 吴志强. 2021. 基于域适应的X光图像的目标检测. 计算机科学, 48(1): 175-181 [DOI: 10.11896/jsjkx.200200023]

Hou Y Y. 2018. Research on the Relationship between Work Stress and Safety Performance of Airport Security Inspectors. Beijing: Beijing Jiaotong University

侯彦伊. 2018. 机场安检人员工作压力与安全绩效关系研究. 北京: 北京交通大学

Jaccard N, Rogers T W, Morton E J and Griffin L D. 2017. Detection of concealed cars in complex cargo X-ray imagery using deep learning. Journal of X-Ray Science and Technology, 25(3): 323-339 [DOI: 10.3233/XST-16199]

Liu K W, Ma Y, Xiong H X, Yan Z J, Zhou Z J, Liu C Y, Fang P P, Li X J and Chen Y L. 2020. Medical-image super-resolution reconstruction method based on residual channel attention network. Laser and Optoelectronics Progress, 57(2): #021014

刘可文, 马圆, 熊红霞, 严泽军, 周志军, 刘朝阳, 房攀攀, 李小军, 陈亚雷. 2020. 基于残差通道注意力网络的医学图像超分辨率重建方法. 激光与光电子学进展, 57(2): #021014 [DOI: 10.3788/LOP57.021014]

Liu Y, Zhang C, Cheng J, Chen X and Wang Z J. 2019. A multi-scale data fusion framework for bone age assessment with convolutional neural networks. Computers in Biology and Medicine, 108: 161-173 [DOI: 10.1016/j.compbiomed.2019.03.015]

Mery D, Svec E, Arias M, Riffo V, Saavedra J M and Banerjee S. 2017. Modern computer vision techniques for X-ray testing in baggage inspection. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(4): 682-692 [DOI: 10.1109/TSMC.2016.2628381]

Miao C J, Xie L X, Wan F, Su C, Liu H Y, Jiao J B and Ye Q X. 2019. SIXray: a large-scale security inspection X-ray benchmark for prohibited item discovery in overlapping images//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE: 2114-2123 [ DOI: 10.1109/CVPR.2019.00222 http://dx.doi.org/10.1109/CVPR.2019.00222 ]

Morris T, Chien T and Goodman E. 2018. Convolutional neural networks for automatic threat detection in security X-ray images//Proceedings of the 17th IEEE International Conference on Machine Learning and Applications (ICMLA). Orlando, USA: IEEE: 285-292 [ DOI: 10.1109/ICMLA.2018.00049 http://dx.doi.org/10.1109/ICMLA.2018.00049 ]

Rubinstein R. 1999. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1(2): 127-190 [DOI: 10.1023/A:1010091220143]

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L and Polosukhin I. 2017. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, USA: Curran Associates Inc.: 6000-6010

Wang F, Jiang M Q, Qian C, Yang S, Li C, Zhang H G, Wang X G and Tang X O. 2017. Residual attention network for image classification//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 6450-6458 [ DOI: 10.1109/CVPR.2017.683 http://dx.doi.org/10.1109/CVPR.2017.683 ]

Wei W Q, Yu J, Bo M Y and Xiao C B. 2021. Video object detection using fusion of SSD and spatiotemporal features. Journal of Image and Graphics, 26(3): 542-555

尉婉青, 禹晶, 柏鳗晏, 肖创柏. 2021. SSD与时空特征融合的视频目标检测. 中国图象图形学报, 26(3): 542-555 [DOI: 10.11834/jig.200020]

Woo S, Park J, Lee J Y and Kweon I S. 2018. CBAM: convolutional block attention module//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 3-19 [ DOI: 10.1007/978-3-030-01234-2_1 http://dx.doi.org/10.1007/978-3-030-01234-2_1 ]

Xiao T J, Xu Y C, Yang K Y, Zhang J X, Peng Y X and Zhang Z. 2015. The application of two-level attention models in deep convolutional neural network for fine-grained image classification//Proceedings of 2015 Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE: 842-850 [ DOI: 10.1109/CVPR.2015.7298685 http://dx.doi.org/10.1109/CVPR.2015.7298685 ]

Zhang W F, Ni J J, Liu L B and Hu Q M. 2021. Symmetric triangle network for object detection within X-ray baggage security imagery//Proceedings of 2021 International Joint Conference on Neural Networks (IJCNN). Shenzhen, China: IEEE: 1-7 [ DOI: 10.1109/IJCNN52387.2021.9533991 http://dx.doi.org/10.1109/IJCNN52387.2021.9533991 ]

Zhao Z H, Zhang H G and Yang J F. 2018. A GAN-based image generation method for X-ray security prohibited items//Proceedings of the 1 st Chinese Conference on Pattern Recognition and Computer Vision. Guangzhou, China: Springer: 420-430 [ DOI: 10.1007/978-3-030-03398-9_36 http://dx.doi.org/10.1007/978-3-030-03398-9_36 ]

Alert me when the article has been cited

提交

Lightweight spaceborne remote sensing object detection algorithm with multi-attention mechanism

Cross-modal feature fusion and detail-enhanced RGB-D salient object detection

Offline recognition of handwritten mathematical expressions： a survey

Two-stage vision transformer for fusing global and local features in distracted driving behavior recognition

Multi-scale attention encoder-decoder network for low-dose dental CBCT denoising