发布时间: 2021-11-16 摘要点击次数: 全文下载次数: DOI: 10.11834/jig.200793 2021 | Volume 26 | Number 11 电力视觉前沿技术

1. 华北电力大学电气与电子工程学院, 保定 071003;
2. 华北电力大学河北省电力物联网技术重点实验室, 保定 071003;
3. 山东大学计算机科学与技术学院, 青岛 266237
 收稿日期: 2020-12-15; 修回日期: 2021-05-12; 预印本日期: 2021-05-19 基金项目: 国家自然科学基金项目（61871182，61773160）；北京市自然科学基金项目（4192055）；河北省自然科学基金项目（F2020502009）；中央高校基本科研业务费专项资金资助（2018MS095，2020YJ006）；模式识别国家重点实验室开放课题基金项目（201900051） 作者简介: 戚银城, 1968年生, 男, 教授, 硕士生导师, 主要研究方向为电力信息分析与智能处理。E-mail: qiych@ncepu.edu.cn 武学良, 男, 硕士研究生, 主要研究方向为目标检测及电网智能巡检技术。E-mail: wuxl1995@163.com 赵振兵, 通信作者, 男, 教授, 博士生导师, 主要研究方向为电力视觉检测。E-mail: zhaozhenbing@ncepu.edu.cn 史博强, 男, 硕士研究生, 主要研究方向为电力信息分析与智能处理及人体行为识别。E-mail: 2968109821@qq.com 聂礼强, 男, 教授, 博士生导师, 主要研究方向为多媒体计算与信息检索。E-mail: nieliqiang@gmail.com *通信作者: 赵振兵  zhaozhenbing@ncepu.edu.cn 中图法分类号: TP181 文献标识码: A 文章编号: 1006-8961(2021)11-2594-11

# 关键词

Bolt defect detection for aerial transmission lines using Faster R-CNN with an embedded dual attention mechanism
Qi Yincheng1,2, Wu Xueliang1, Zhao Zhenbing1,2, Shi Boqiang1, Nie Liqiang3
1. School of Electrical and Electronic Engineering, North China Electric Power University, Baoding 071003, China;
2. Hebei Key Laboratory of Power Internet of Things Technology, North China Electric Power University, Baoding 071003, China;
3. School of Computer Science and Technology, Shandong University, Qingdao 266237, China
Supported by: National Natural Science Foundation of China (61871182, 61773160); Beijing Municipal Natural Science Foundation (4192055); Natural Science Foundation of Hebei Province, China (F2020502009); Fundamental Research Funds for the Central Universities (2018MS095, 2020YJ006); Open Project Program of the National Laboratory of Pattern Recognition(NLPR) (201900051)

# Abstract

Objective In transmission lines, bolts are widely used as a kind of fasteners to connect various parts of transmission lines and make the overall structure stable and safe. However, bolts are easily damaged because of their complex working environment. The damage or loss of a bolt may cause a large area of transmission line failure, which seriously threatens the safety and stability of the power grid. Bolts are the most common components of transmission lines. Thus, bolt defect detection is an important task in transmission line inspection. Good features are difficult extract because of the complex background, small target, small difference between categories, and loss of gradient information. This study proposes a dual-attention scheme to enhance the visual features of different scales and positions. Method First, for different scales, the network extracts the feature map of each layer, uses the multi-scale attention model to obtain the corresponding attention map, calculates the difference of the attention map for adjacent layers, and adds it to the loss function as a regularization term to enhance the fine features of the bolt area. The trained network continuously reduces the difference in the attention maps of different layers. The learned attention maps of different scales are introduced into the network as a kind of context information. This procedure can avoid the loss of important information in the process of feature extraction. No additional regulatory information is required because the attention map is from the network itself. Second, for different positions, bolts appear in specific positions of the accessories, but due to light blocking and other reasons, the characteristics of these positions are not obvious. In this study, we use the feature map to derive a spatial attention map of the image. Each element in the attention map indicates the degree of similarity between two spatial locations. Then, the attention map is used to combine the features of each position with the global feature. This process enhances the features in similar regions and improves the difference degree between dissimilar areas. Hence, the difference between the bolt and the background is increased, and the detection accuracy of the bolt area is improved. Result The method is tested on a typical bolt data set for aerial transmission lines. The typical bolt data set contains 1 483 images of three types of bolts. Each image has a size of approximately 3 000×4 000 pixels. A total of 2 692 targets are labeled, and they include 1 443 normal bolt samples, 670 missing bolt samples, and 579 missing nut bolt samples. The ratio of the training set to the test set is 8:2. The baseline model used in this study is the faster region convolutional neural network(Faster R-CNN) model. Experimental results show that compared with the baseline, the proposed model's mean average precision (mAP) is increased by 0.29% when the multi-scale attention module is added. Normal, missing and missing nut bolts increase by 0.62%, 2.54%, and 0.69%, respectively. After the addition of the spatial attention module, the mAP of the model increases by 0.61%; specifically, the AP of normal bolts increases by 0.3%, that of missing bolts increases by 2.05%, and that of missing nut bolts increases by 0.52%. This result is obtained because several shaded nuts of missing bolts are confused with the nuts of normal bolts, leading to misjudgment. After introducing multi-scale attention and spatial attention at the same time, the model's mAP is increased by 2.21%; the AP of the normal, missing, and missing nut bolts is increased by 0.29%, 5.23%, and 1.10%, respectively. These experimental results prove the effectiveness of the bolt defect detection method for aerial transmission lines based on the dual attention mechanism. This study also conducts visualization experiments, including the establishment of feature maps, model training loss function curve, precision-recall(PR) curve, and bolt defect detection result map, to prove that the proposed method can be applied to feature extraction. Conclusion Experimental results prove that the proposed detection method for aerial transmission line bolt defects based on the dual attention mechanism is effective. The process of supervising feature extraction can ensure that abundant useful information is retained when extracting features. For the bolt defect detection task, increasing the difference between the target and the background can improve the detection accuracy of the target area. The visualization experiments verify that the proposed method can retain abundant useful information in the process of feature extraction. The visualized test examples also prove that the proposed method can effectively avoid the problem of misjudgment in bolt defect detection.

# Key words

dual attention mechanism; multi-scale; spatial position; bolt defect detection; deep learning

# 1.1 多尺度注意力

 $P_{s} =\sum\limits_{i=1}^{C_{k}}\left|A_{k i}\right|$ (1)

 $P_{s}^{2} =\sum\limits_{i=1}^{C_{k}}\left|A_{k i}\right|^{2}$ (2)

 $P_{\max }^{2} =\max \limits_{i=1, \cdots, C}\left|A_{k i}\right|^{2}$ (3)

 $\varphi\left(\boldsymbol{A}_{k}\right)={softmax}\left({ upsampling }\left(P_{s}^{2}\left(\boldsymbol{A}_{k}\right)\right)\right)$ (4)

 $L_{k, k+1}\left(\boldsymbol{A}_{k}, \boldsymbol{A}_{k+1}\right)=L_{2}\left(\varphi\left(\boldsymbol{A}_{k}\right), \varphi\left(\boldsymbol{A}_{k+1}\right)\right)$ (5)

 $L_{\mathrm{sum}}=L_{1,2}+L_{2,3}$ (6)

# 1.2 空间注意力

 $\boldsymbol{E} =f\left(\boldsymbol{A}_{3}\right)$ (7)

 $\boldsymbol{F} =f\left(\boldsymbol{A}_{3}\right)$ (8)

 $\boldsymbol{S}={softmax}\left(\boldsymbol{E}^{\prime \mathrm{T}} \cdot \boldsymbol{F}^{\prime}\right)$ (9)

 $s_{j i}=\frac{\exp \left(\boldsymbol{E}_{i}^{\prime \mathrm{T}} \cdot \boldsymbol{F}_{j}^{\prime}\right)}{\sum\limits_{i=1}^{N} \exp \left(\boldsymbol{E}_{i}^{\prime \mathrm{T}} \cdot \boldsymbol{F}_{j}^{\prime}\right)}$ (10)

 $T_{j}=\alpha \sum\limits_{i=1}^{N} s_{j i} G_{i}^{\prime}+A_{3 j}$ (11)

# 1.3 损失函数

 $L o s s=L(y, \hat{y})+\lambda L_{\mathrm{sum}}$ (12)

# 2.2 实验结果与分析

 $P =\frac{T P}{T P+F P}$ (13)

 $R =\frac{T P}{T P+F N}$ (14)

 $m A P =\frac{\sum\limits_{i=1}^{N_{\mathrm{cls}}} \int_{0}^{1} P_{i}\left(R_{i}\right) \mathrm{d} R}{N_{\mathrm{cls}}}$ (15)

Table 1 Ablation experiment

 /% 模型 AP mAP 正常螺栓 缺销螺栓 螺母缺失螺栓 基线 86.35 66.12 87.06 79.84 基线+多尺度注意力 86.97 68.66 87.75 81.13 基线+空间注意力 86.65 68.17 86.54 80.45 本文 86.64 71.35 88.16 82.05

Table 2 mAP with different $\lambda$ value

 /% $\lambda$ 0.001 0.005 0.01 0.05 0.1 0.5 0.6 0.7 0.8 0.9 1.0 mAP 79.42 78.92 82.05 78.80 78.98 77.98 79.08 81.49 80.74 80.73 79.59

# 3 结论

1) 为了避免螺栓图像在特征提取阶段由于螺栓目标小、背景复杂等问题导致的信息丢失，本文提出利用多尺度注意力模块对特征提取阶段进行监督，同时用于监督的注意力图来自于网络本身，因此不需要额外的外部监督信息。

2) 为了解决检测过程中螺栓区域检测效果不佳的问题，本文提出利用空间注意力模块对螺栓图像在空间上加权，相似特征互相增强，提高螺栓区域与背景区域的差异程度。

# 参考文献

• Chen J W, Liu Z G, Wang H R, Núñez A, Han Z W. 2018. Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 67(2): 257-269 [DOI:10.1109/TIM.2017.2775345]
• Everingham M, Van Gool L, Williams C K I, Winn J, Zisserman A. 2010. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2): 303-338 [DOI:10.1007/s11263-009-0275-4]
• Han K, Sun M Y, Zhou X G, Zhang G H, Dang H and Liu Z C. 2017. A new method in wheel hub surface defect detection: object detection algorithm based on deep learning//Proceedings of 2017 International Conference on Advanced Mechatronic Systems. Xiamen, China: IEEE: 335-338[DOI: 10.1109/ICAMechS.2017.8316494]
• He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 770-778[DOI: 10.1109/CVPR.2016.90]
• Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollá P and Zitnick C L. 2014. Microsoft COCO: common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 740-755[DOI: 10.1007/978-3-319-10602-1_48]
• Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y and Berg A C. 2016. SSD: single shot multibox detector//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 21-37[DOI: 10.1007/978-3-319-46448-0_2]
• Liu Z G, Zhong J P, Lyu Y, Liu K, Han Y, Wang L Y and Liu W Q. 2018. Location and fault detection of catenary support components based on deep learning//Proceedings of 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC). Houston, USA: IEEE: 1-6[DOI: 10.1109/I2MTC.2018.8409637]
• Mei S, Yang H, Yin Z Q. 2018. An unsupervised-learning-based approach for automated defect inspection on textured surfaces. IEEE Transactions on Instrumentation and Measurement, 67(6): 1266-1277 [DOI:10.1109/TIM.2018.2795178]
• Nguyen V N, Jenssen R, Roverso D. 2018. Automatic autonomous vision-based power line inspection: a review of current status and the potential role of deep learning. International Journal of Electrical Power and Energy Systems, 99: 107-120 [DOI:10.1016/j.ijepes.2017.12.016]
• Qi Y C, Lang J Y, Zhao Z B, Jiang A X, Nie L Q. 2019. Relativistic GAN for bolts image generation with attention mechanism. Electrical Measurement and Instrumentation, 56(19): 64-69 (戚银城, 郎静宜, 赵振兵, 江爱雪, 聂礼强. 2019. 结合注意力机制的相对GAN螺栓图像生成. 电测与仪表, 56(19): 64-69) [DOI:10.19753/j.issn1001-1390.2019.019.011]
• Qiao L, Zhao E D, Liu J J, Cheng B. 2017. Research of workpiece defect detection method based on CNN. Computer Science, 44(S11): 238-243 (乔丽, 赵尔敦, 刘俊杰, 程彬. 2017. 基于CNN的工件缺陷检测方法研究. 计算机科学, 44(S11): 238-243) [DOI:10.11896/j.issn.1002-137X.2017.11A.050]
• Redmon J, Divvala S, Girshick R and Farhadi A. 2016. You only look once: unified, real-time object detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 779-788. [DOI: 10.1109/CVPR.2016.91]
• Ren S Q, He K M, Girshick R, Sun J. 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 1137-1149 [DOI:10.1109/TPAMI.2016.2577031]
• Tang Y, Han J, Wei W L, Ding J, Peng X J. 2018. Research on part recognition and defect detection of trainsmission line in deep learning. Electronic Measurement Technology, 41(6): 60-65 (汤踊, 韩军, 魏文力, 丁建, 彭新俊. 2018. 深度学习在输电线路中部件识别与缺陷检测的研究. 电子测量技术, 41(6): 60-65) [DOI:10.19651/j.cnki.emt.1701266]
• Zhao Z B, Qi H Y, Nie L Q. 2019. Research overview on visual detection of transmission lines based on deep learning. Guangdong Electric Power, 32(9): 11-23 (赵振兵, 齐鸿雨, 聂礼强. 2019. 基于深度学习的输电线路视觉检测研究综述. 广东电力, 32(9): 11-23) [DOI:10.3969/j.issn.1007-290X.2019.009.002]
• Zhao Z B, Li Y X, Qi Y C, Kong Y H, Nie L Q. 2020. Insulator defect detection method based on dynamic focus loss function and sample balance method. Electric Power Automation Equipment, 40(10): 205-211 (赵振兵, 李延旭, 戚银城, 孔英会, 聂礼强. 2020. 基于动态焦点损失函数和样本平衡方法的绝缘子缺陷检测方法. 电力自动化设备, 40(10): 205-211) [DOI:10.16081/j.epae.202010008]
• Zhao Z B, Qi H Y, Qi Y C, Zhang K, Zhai Y J, Zhao W Q. 2020. Detection method based on automatic visual shape clustering for pin-missing defect in transmission lines. IEEE Transactions on Instrumentation and Measurement, 69(9): 6080-6091 [DOI:10.1109/TIM.2020.2969057]
• Zhong J Q, Liu Z G, Han Z W, Han Y, Zhang W X. 2018. A CNN-based defect inspection method for catenary split pins in high-speed railway. IEEE Transactions on Instrumentation and Measurement, 68(8): 2849-2860 [DOI:10.1109/TIM.2018.2871353]