张珂,何颖宣,赵凯,冯晓晗,赵振兵,马占宇(华北电力大学电子与通信工程系, 保定 071003;华北电力大学河北省电力物联网技术重点实验室, 保定 071003;北京邮电大学人工智能学院, 北京 100086)
目的 螺栓是确保输电线路安全牢靠的基石，螺栓缺销、松动和锈蚀等缺陷是造成输电线路损坏甚至重大事故的重要原因之一。本文针对螺栓缺陷存在视觉不可分的问题，提出了一种改进NTS-Net（navigator-teacher-scrutinizer network）的螺栓属性多标签分类方法。方法 为了增强模型对不规则轮廓的处理能力，采用可变形卷积的ResNet-50网络作为特征提取网络从原图中提取全局特征。利用NTS-Net学习得到图像信息量最大的判别性局部区域。为了考虑不同局部特征对不同属性标签的不同影响，在局部特征与全局特征融合时引入通道注意力机制，提取特征的通道权重，获取关键通道特征以改善多标签分类效果。结果 实验结果表明，本文在螺栓多属性分类数据集上的平均分类精确率为84.5%，比采用传统的多标签分类精确率提升了10%~20%。结论 本文通过可变形卷积提升网络的特征提取能力以及引入通道注意力机制实现了对NTS-Net提供的局部特征的高效利用，为解决螺栓多属性分类中存在的问题提供了一种新的思路。
Multi-label classification method of bolt attributes based on deformable NTS-Net
Zhang Ke,He Yingxuan,Zhao Kai,Feng Xiaohan,Zhao Zhenbing,Ma Zhanyu(Department of Electronic and Communication Engineering, North China Electric Power University, Baoding 071003, China;Hebei Key Laboratory of Power Internet of Things Technology, North China Electric Power University, Baoding 071003, China;School of Artificial Intelligence, Beijing University of Posts and Telecommunication, Beijing 100086, China)
Objective Unmanned aerial vehicle(UAV)-based transmission line inspection technology has achieved long-term progress and development. The use of computer vision technology to automatically and accurately locate line equipment, such as wires, insulators, and bolts, from aerial inspection images under complex natural backgrounds and accurately detect their defects has become animportant technical issue. The defects inspected by transmission lines mainly include tower, wire, insulator,and metal fitting defects. Given the large size of metal fittings and insulators, their defects are obvious and easy to identify. By contrast, numerous bolts are present in poles, insulators, and metal fittings. Bolts change easily from the normal state to the defect state due to the large number of bolts and complex stress conditions. The use of deep learning has achieved good results in visual detection, identification, and classification of tower, wire, insulator,and metal fitting defects, but only a few studies have been conducted on bolt defects. In addition, bolt defects are not completely visually separable problems; they are visually inseparable, and they can not be solved by object detection algorithms alone. Thus, we believe that the bolt defect detection problem is not only an object detection problem, but also an image classification problem. Multi-label classification of bolts must be implemented efficiently and quickly to provide a basis for defect detection. The convolutional neural network (CNN) is inherently limited by model geometry transformation due to its fixed geometric structure. An offset variable must be added to the position of each sampling point in the convolution kernel to weaken this limitation and improve the feature extraction capability of bolts. By adding these variables, the convolution kernel is given random sampling near the current position and is no longer limited to the previous regular grid points. The convolution operation after expansion is called deformable convolution. Deformable convolution changes the sampling position of the standard convolution kernel by adding an additional offset to the sampling point. The compensation obtained can be learned through training without additional supervision.Method The object to be inspected in the transmission line bolt multi-label classification task has similar overall characteristics as those of the object in the general image multi-label classification task. The classification model needs to capture the key local features that can distinguish the attributes of different categories. The idea of using local regions to assist in classification belongs to fine-grained classification. Several studies on fine-grained classification algorithms used detailed local area labels to train the model so that the model can accurately locate the regions containing detailed semantic information. However, this approach requires a huge amount of work in the production of labels. In other studies, unsupervised learning was used to locate key areas. Although this strategy eliminates tedious label-making work, the accuracy of the model in locating key details can not be ensured. The multi-label classification method proposed in this study is mainly divided into three steps.First, navigator-teacher-scrutinizer network(NTS-Net) is used as the basic network, and the feature extraction network is improved into a deformable ResNet-50 network in accordance with the various properties of the bolt target shape. Second, the navigator network in NTS-Net continuously learns and provides k regions with the most information under the guidance of the teacher network to obtain the discriminative region of the bolt target. Lastly, to make the model use discriminant features effectively, the input features of the k regions receiving the most information from the navigator network are extracted, and corresponding feature vectors are generated and connected to the feature vectors of the entire input image. Afterward, the features need to be passed through the channel attention module, which can enhance the feature with a large weight and suppress the feature with a small weight. Result This study uses the bolt multi-attribute classification dataset to evaluate the model. The bolt defect images are from samples obtained by UAV line inspection. The data sample has a total of 2 000 pictures, of which 1 500 are used as training samples and 500 are used as test samples. The bolt defect attributes are divided into six categories based the idea of visual separability. Each bolt defect image contains one or more defect attributes, which can be divided into the following six categories:a pin hole is present, shim is present, a nut is present, rust is present, the nut is loose, and the pin is loose; they are labeled 0-5 respectively. In the multi-label classification task in this study, a 1×6 matrix is constructed for each picture as the label of the picture. If the corresponding attribute category exists, the value is set to 1 and vice versa. Experimental results show that the mean average precision of the proposed method in the bolt multi-attribute classification dataset is 84.5%, which is 10%~20% higher than the accuracy of multi-label classification using traditional networks. Conclusion The feature extraction capability of the network is improved through deformable convolution, and the channel attention mechanism is introduced to realize the efficient utilization of the local features provided by NTS-Net. Experimental results show that the proposed method performs better than the traditional method in the bolt multi-attribute classification dataset. The proposed method provides a new idea for applying multi-attribute information to bolt defect reasoning and realizing bolt defect detection.