Bolt defect detection for aerial transmission lines using Faster R-CNN with an embedded dual attention mechanism

Yincheng Qi; Xueliang Wu; Zhenbing Zhao; Boqiang Shi; Liqiang Nie

doi:10.11834/jig.200793

Frontier Technology of Power Computer Vision | Views : 0 下载量: 63 CSCD: 16

PDF
Export
Share
Collection
Album

Bolt defect detection for aerial transmission lines using Faster R-CNN with an embedded dual attention mechanism
Vol. 26, Issue 11, Pages: 2594-2604(2021)
Received：15 December 2020，

Revised：2021-5-12，

Accepted：19 May 2021，

Published：16 November 2021
DOI： 10.11834/jig.200793
稿件说明：

移动端阅览

DOI：

Yincheng Qi, Xueliang Wu, Zhenbing Zhao, Boqiang Shi, Liqiang Nie. Bolt defect detection for aerial transmission lines using Faster R-CNN with an embedded dual attention mechanism[J]. Journal of Image and Graphics, 2021, 26(11): 2594-2604. DOI： 10.11834/jig.200793.

摘要

目的

螺栓是输电线路上最普遍、数量最多的部件，螺栓缺陷检测是输电线路视觉检测任务的一大难点。针对螺栓目标存在背景复杂、目标过小、不同类别之间差异小以及精细特征难以提取的问题，提出一种双注意力机制方法，分别对不同尺度和不同位置的视觉特征进行分析和增强。

方法

对于不同尺度的特征，网络使用不同尺度的特征图计算出相应的注意力图，然后将相邻层的注意力图差异性作为正则化项加入网络中，从而增强螺栓区域的精细特征。对于不同位置的特征，先使用特征图计算出图像的空间注意力图，注意力图中每个元素表示两个空间位置的相似程度，然后利用注意力图将局部特征与全局特征融合，从而在全局视野上增强相似的区域，达到增大螺栓与背景的特征差异程度，实现提高螺栓区域的预测效果。

结果

本文在航拍输电线路典型螺栓数据集上进行测试，与基线相比，结合双注意力机制的航拍输电线路螺栓检测方法的平均准确率提高了2.21%，其中正常螺栓类提升了0.29%，缺销螺栓类提升了5.23%，螺母缺失螺栓类提升了1.1%。

结论

本文提出的基于双注意力机制的航拍输电线路螺栓缺陷检测方法取得了良好的效果，有效避免了螺栓缺陷检测中的误判漏判问题，为进一步对输电线路其他缺陷任务奠定了良好的基础。

Abstract

Objective

In transmission lines

bolts are widely used as a kind of fasteners to connect various parts of transmission lines and make the overall structure stable and safe. However

bolts are easily damaged because of their complex working environment. The damage or loss of a bolt may cause a large area of transmission line failure

which seriously threatens the safety and stability of the power grid. Bolts are the most common components of transmission lines. Thus

bolt defect detection is an important task in transmission line inspection. Good features are difficult extract because of the complex background

small target

small difference between categories

and loss of gradient information. This study proposes a dual-attention scheme to enhance the visual features of different scales and positions.

Method

First

for different scales

the network extracts the feature map of each layer

uses the multi-scale attention model to obtain the corresponding attention map

calculates the difference of the attention map for adjacent layers

and adds it to the loss function as a regularization term to enhance the fine features of the bolt area. The trained network continuously reduces the difference in the attention maps of different layers. The learned attention maps of different scales are introduced into the network as a kind of context information. This procedure can avoid the loss of important information in the process of feature extraction. No additional regulatory information is required because the attention map is from the network itself. Second

for different positions

bolts appear in specific positions of the accessories

but due to light blocking and other reasons

the characteristics of these positions are not obvious. In this study

we use the feature map to derive a spatial attention map of the image. Each element in the attention map indicates the degree of similarity between two spatial locations. Then

the attention map is used to combine the features of each position with the global feature. This process enhances the features in similar regions and improves the difference degree between dissimilar areas. Hence

the difference between the bolt and the background is increased

and the detection accuracy of the bolt area is improved.

Result

The method is tested on a typical bolt data set for aerial transmission lines. The typical bolt data set contains 1 483 images of three types of bolts. Each image has a size of approximately 3 000×4 000 pixels. A total of 2 692 targets are labeled

and they include 1 443 normal bolt samples

670 missing bolt samples

and 579 missing nut bolt samples. The ratio of the training set to the test set is 8:2. The baseline model used in this study is the faster region convolutional neural network(Faster R-CNN) model. Experimental results show that compared with the baseline

the proposed model's mean average precision (mAP) is increased by 0.29% when the multi-scale attention module is added. Normal

missing and missing nut bolts increase by 0.62%

2.54%

and 0.69%

respectively. After the addition of the spatial attention module

the mAP of the model increases by 0.61%; specifically

the AP of normal bolts increases by 0.3%

that of missing bolts increases by 2.05%

and that of missing nut bolts increases by 0.52%. This result is obtained because several shaded nuts of missing bolts are confused with the nuts of normal bolts

leading to misjudgment. After introducing multi-scale attention and spatial attention at the same time

the model's mAP is increased by 2.21%; the AP of the normal

missing

and missing nut bolts is increased by 0.29%

5.23%

and 1.10%

respectively. These experimental results prove the effectiveness of the bolt defect detection method for aerial transmission lines based on the dual attention mechanism. This study also conducts visualization experiments

including the establishment of feature maps

model training loss function curve

precision-recall(PR) curve

and bolt defect detection result map

to prove that the proposed method can be applied to feature extraction.

Conclusion

Experimental results prove that the proposed detection method for aerial transmission line bolt defects based on the dual attention mechanism is effective. The process of supervising feature extraction can ensure that abundant useful information is retained when extracting features. For the bolt defect detection task

increasing the difference between the target and the background can improve the detection accuracy of the target area. The visualization experiments verify that the proposed method can retain abundant useful information in the process of feature extraction. The visualized test examples also prove that the proposed method can effectively avoid the problem of misjudgment in bolt defect detection.

关键词

Keywords

references

Chen J W, Liu Z G, Wang H R, Núñez A and HanZ W. 2018. Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 67(2): 257-269[DOI: 10.1109/TIM.2017.2775345]

Everingham M, Van Gool L, Williams C K I, Winn J and Zisserman A. 2010. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2): 303-338[DOI: 10.1007/s11263-009-0275-4]

Han K, Sun M Y, Zhou X G, Zhang G H, Dang H and Liu Z C. 2017. A new method in wheel hub surface defect detection: object detection algorithm based on deep learning//Proceedings of 2017 International Conference on Advanced Mechatronic Systems. Xiamen, China: IEEE: 335-338[ DOI: 10.1109/ICAMechS.2017.8316494 http://dx.doi.org/10.1109/ICAMechS.2017.8316494 ]

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, DolláP and Zitnick C L. 2014. Microsoft COCO: common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 740-755[ DOI: 10.1007/978-3-319-10602-1_48 http://dx.doi.org/10.1007/978-3-319-10602-1_48 ]

Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y and Berg A C. 2016. SSD: single shot multibox detector//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 21-37[ DOI: 10.1007/978-3-319-46448-0_2 http://dx.doi.org/10.1007/978-3-319-46448-0_2 ]

Liu Z G, Zhong J P, Lyu Y, Liu K, Han Y, Wang L Y and Liu W Q. 2018. Location and fault detection of catenary support components based on deep learning//Proceedings of 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC). Houston, USA: IEEE: 1-6[ DOI: 10.1109/I2MTC.2018.8409637 http://dx.doi.org/10.1109/I2MTC.2018.8409637 ]

Mei S, Yang H and Yin Z Q. 2018. An unsupervised-learning-based approach for automated defect inspection on textured surfaces. IEEE Transactions on Instrumentation and Measurement, 67(6): 1266-1277[DOI: 10.1109/TIM.2018.2795178]

Nguyen V N, Jenssen R and Roverso D. 2018. Automatic autonomous vision-based power line inspection: a review of current status and the potential role of deep learning. International Journal of Electrical Power and Energy Systems, 99: 107-120[DOI: 10.1016/j.ijepes.2017.12.016]

Qi Y C, Lang J Y, Zhao Z B, Jiang A X and Nie L Q. 2019. Relativistic GAN for bolts image generation with attention mechanism. Electrical Measurement and Instrumentation, 56(19): 64-69

戚银城, 郎静宜, 赵振兵, 江爱雪, 聂礼强. 2019. 结合注意力机制的相对GAN螺栓图像生成. 电测与仪表, 56(19): 64-69 [DOI: 10.19753/j.issn1001-1390.2019.019.011]

Qiao L, Zhao E D, Liu J J and Cheng B. 2017. Research of workpiece defect detection method based on CNN. Computer Science, 44(S11): 238-243

乔丽, 赵尔敦, 刘俊杰, 程彬. 2017. 基于CNN的工件缺陷检测方法研究. 计算机科学, 44(S11): 238-243 [DOI: 10.11896/j.issn.1002-137X.2017.11A.050]

Redmon J, Divvala S, Girshick R and Farhadi A. 2016. You only look once: unified, real-time object detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 779-788. [ DOI: 10.1109/CVPR.2016.91 http://dx.doi.org/10.1109/CVPR.2016.91 ]

Ren S Q, He K M, Girshick R and Sun J. 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 1137-1149[DOI: 10.1109/TPAMI.2016.2577031]

Tang Y, Han J, Wei W L, Ding J and Peng X J. 2018. Research on part recognition and defect detection of trainsmission line in deep learning. Electronic Measurement Technology, 41(6): 60-65

汤踊, 韩军, 魏文力, 丁建, 彭新俊. 2018. 深度学习在输电线路中部件识别与缺陷检测的研究. 电子测量技术, 41(6): 60-65 [DOI: 10.19651/j.cnki.emt.1701266]

Zhao Z B, Qi H Y and Nie L Q. 2019. Research overview on visual detection of transmission lines based on deep learning. Guangdong Electric Power, 32(9): 11-23

赵振兵, 齐鸿雨, 聂礼强. 2019. 基于深度学习的输电线路视觉检测研究综述. 广东电力, 32(9): 11-23 [DOI: 10.3969/j.issn.1007-290X.2019.009.002]

Zhao Z B, Li Y X, Qi Y C, Kong Y H and Nie L Q. 2020. Insulator defect detection method based on dynamic focus loss function and sample balance method. Electric Power Automation Equipment, 40(10): 205-211

赵振兵, 李延旭, 戚银城, 孔英会, 聂礼强. 2020. 基于动态焦点损失函数和样本平衡方法的绝缘子缺陷检测方法. 电力自动化设备, 40(10): 205-211 [DOI: 10.16081/j.epae.202010008]

Zhao Z B, Qi H Y, Qi Y C, Zhang K, Zhai Y J and Zhao W Q. 2020. Detection method based on automatic visual shape clustering for pin-missing defect in transmission lines. IEEE Transactions on Instrumentation and Measurement, 69(9): 6080-6091[DOI: 10.1109/TIM.2020.2969057]

Zhong J Q, Liu Z G, Han Z W, Han Y and Zhang W X. 2018. A CNN-based defect inspection method for catenary split pins in high-speed railway. IEEE Transactions on Instrumentation and Measurement, 68(8): 2849-2860[DOI:10.1109/TIM.2018.2871353]

Alert me when the article has been cited

提交

Adaptive multi-scale super-resolution reconstruction for lunar images captured by a narrow-angle camera

Ultrasonic image segmentation of thyroid nodules-relevant multi-scale feature based h-shape network

Dual attention mechanism based pavement crack detection

Attention mechanism embedded multi-scale restoration method for blurred image