浏览全部资源
扫码关注微信
Received：22 August 2020，
Revised：2021-1-22，
Accepted：29 January 2021，
Published：16 July 2021
移动端阅览
目的
2
借助深度学习强大的识别与检测能力，辅助人工进行电力场景下的危险描述与作业预警是一种较为经济和高效的电力安全监管手段。然而，目前主流的以目标检测技术为基础的预警系统只能给出部分危险目标的信息，忽视了电力设备的单目危险关系和成对对象间潜在的二元危险关系。不同于以往的方法，为了拓展危险预警模块的识别能力与功能范畴，本文提出了一种在电力场景下基于视觉关系检测的自动危险预警描述生成方法。
方法
2
对给定的待检测图像，通过目标检测模块得到图中对象的类别名称和限界框位置；分别对图像进行语义特征、视觉特征和空间位置特征的抽取，将融合后的总特征送入关系检测模块，输出单个对象的一元关系和成对对象间的关系三元组；根据检测出的对象类别和关系信息，进行危险预测并给出警示描述。
结果
2
本文自主搜集了多场景下的电力生产作业图像并进行标注，同时进行大量消融实验。实验显示，结合了语义特征、空间特征和视觉特征的关系检测器在前5召回率Recall@5和前10召回率Recall@10上的精度分别达到86.80%和93.93%，比仅使用视觉特征的关系检测器的性能提高约15%。
结论
2
本文提出的融合多模态特征输入的视觉关系检测网络能够较好地给出谓词关系的最佳匹配，并减少不合理的关系预测，且具有一定零样本学习（zero-shot learning）能力。相关可视化结果表明，整体系统能够较好地完成电力场景下的危险预警描述任务。
Objective
2
The past decade has seen a steady increase in deep learning areas
where extensive research has been published to improve the learning capabilities of deep neural networks. Thus
a growing number of regulators in the electric power industry utilize such deep learning techniques with powerful recognition and detection capabilities to build their surveillance systems
which greatly reduce the risk of major accidents in daily work. However
most of the current early-warning systems are based on object detection technologies
which can only provide annotations of dangerous targets within the image
ignoring the significant information about unary relationships of electrical equipment and binary relationships between paired objects. This condition limits the capabilities of emergency recognition and forewarning. With the presence of powerful object detectors such as Faster region convolutional neural network (R-CNN) and huge visual datasets such as visual genome
visual relationship detection has attracted much attention in recent years. By utilizing the basic building blocks for single-object detection and understanding
visual relationship detection aims to not only accurately localize a pair of objects but also precisely determine the predicate between them. As a mid-level learning task
visual relationship detection can capture the detailed semantics of visual scenes by explicitly modeling objects along with their relationships with other objects. This approach bridges the gap between low-level visual tasks and high-level vision-language tasks
as well as helps machines to solve more challenging visual tasks such as image captioning
visual question answering
and image generation. However
the difficulty is in developing robust algorithms to recognize relationships between paired objects with challenging factors
such as highly diverse visual features in the same predicate category
incomplete annotation and long-tailed distribution in the dataset
and optimum predicate matching problem. Although numerous methods have been proposed to build efficient relationship detectors
few of them concentrate on applying detection technologies to actual use.
Method
2
Different from existing methods
our method introduces the visual relationship detection technology into current early-warning systems. Specifically
our method not only identifies dangerous objects but also recognizes the potential unary or binary relationships that may cause an accident. To sum up
we propose a two-stage emergency recognition and forewarning system for the electric power industry. The system consists of a pre-trained object-detection module and a relationship detection module. The pipeline of our system mainly includes three stages. First
we train an object-detection module based on Faster R-CNN in advance. When given an image
the pre-trained object detector localizes all the object bounding boxes and annotates their categories. Then
the relationship-detection module integrates multiple cues (visual appearance
spatial location
and semantic embedding) to compute the predicate confidence of all the object pairs
and output the top instances as the relationship predictions. Finally
based on the targets and relationship information provided by the detectors
our system performs emergency prediction and generates a warning description that may help regulators in the electric power industry to make suitable decisions.
Result
2
We conduct several experiments to prove the efficiency and superiority of our method. First
we collect and build a dataset consisting of large amounts of images from multiple scenarios in the electric power industry. Using instructions from experts
we define and label the relationship categories that may pose risks to the images in the dataset. Then
according to the number of objects forming a relationship
we divide the dataset into two parts. Thus
our experiments involve two relevant tasks to evaluate the proposed method: unary relationship detection and binary relationship detection. For the unary relationship detection
we use precision and recall as thee valuation metrics. For the binary relationship detection
the evaluation metrics are Recall@5 and Recall@10. As our proposed relationship-detection module contains multiple cues to learn the holistic representation of a relationship instance
we conduct ablation experiments to explore their influence on the final performance. Experiment results show that the detector that uses visual
spatial
and semantic features as input achieve the best performance of 86.80% in Recall@5 and 93.93% in Recall@10.
Conclusion
2
Extensive experiments show that our proposed method is efficient and effective in detecting defective electrical equipment and dangerous relationships between paired objects. Moreover
we formulate a pre-defined rule to generate the early-warning description according to the results of the object and relationship detectors. All of the proposed methods can help regulators take proper and timely actions to avoid harmful accidents in the electric power industry.
Dai B, Zhang Y Q and Lin D H. 2017. Detecting visual relationships with deep relational networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 3298-3308[ DOI: 10.1109/cvpr.2017.352 http://dx.doi.org/10.1109/cvpr.2017.352 ]
Deng J, Dong W, Socher R, Li L J, Li K and Li F F. 2009. ImageNet: a large-scale hierarchical image database//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, USA: IEEE: 248-255[ DOI: 10.1109/CVPR.2009.5206848 http://dx.doi.org/10.1109/CVPR.2009.5206848 ]
Gella S, Lapata M and Keller F. 2016. Unsupervised visual sense disambiguation for verbs using multimodal embeddings[EB/OL]. [2020-07-22] . https://arxiv.org/pdf/1603.09188.pdf https://arxiv.org/pdf/1603.09188.pdf
Girshick R, Donahue J, Darrell T and Malik J. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE: 580-587[ DOI: 10.1109/CVPR.2014.81 http://dx.doi.org/10.1109/CVPR.2014.81 ]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Hu H, Gu J Y, Zhang Z, Dai J F and Wei Y C. 2018. Relation networks for object detection//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 3588-3597[ DOI: 10.1109/CVPR.2018.00378 http://dx.doi.org/10.1109/CVPR.2018.00378 ]
Hu Z W. 2016. Discussion on safety hazard identification and management methods of transmission line projects. China High-Tech Enterprises, (32): 131-132
胡正文. 2016. 输电线路工程的安全危险辨识及管理方法探讨. 中国高新技术企业, (32): 131-132 [DOI:10.13535/j.cnki.11-4406/n.2016.32.065]
Hwang S J, Kim H J, Ravi S N, Collins M D, Tao Z R and Singh V. 2018. Tensorize, factorize and regularize: robust visual relationship learning//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 1014-1023[ DOI: 10.1109/CVPR.2018.00112 http://dx.doi.org/10.1109/CVPR.2018.00112 ]
Kingma D P andBa J L. 2014. Adam: a method for stochastic optimization[EB/OL]. [2020-07-22] . https://arxiv.org/pdf/1412.6980.pdf https://arxiv.org/pdf/1412.6980.pdf
Krishna R, Chami I, Bernstein M and Li F F. 2018. Referring relationships//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 6867-6876[ DOI: 10.1109/CVPR.2018.00718 http://dx.doi.org/10.1109/CVPR.2018.00718 ]
Liang K M, Guo Y H, Chang H and Chen X L. 2018. Visual relationship detection with deep structural ranking//Proceedings of AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI: 7098-7105
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y and Berg A C. 2016. SSD: single shot multibox detector//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 21-37[ DOI: 10.1007/978-3-319-46448-0_2 http://dx.doi.org/10.1007/978-3-319-46448-0_2 ]
Liu W, Huang S, Ma K and Chen H. 2014. Application of video monitoring system in power system. Guangdong Electric Power, 27(4): 57-60
刘玮, 黄曙, 马凯, 陈皓. 2014. 视频监控技术在电力系统中的应用. 广东电力, 27(4): 57-60 [DOI:10.3969/j.issn.1007-290X.2014.04.012]
Lu C W, Krishna R, Bernstein M and Li F F. 2016. Visual relationship detection with language priors//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, the Netherlands: Springer: 852-869[ DOI: 10.1007/978-3-319-46448-0_51 http://dx.doi.org/10.1007/978-3-319-46448-0_51 ]
Lu J D. 2017. Fault Diagnosis Expert System of Power Equipment Based on Deep Learning. Beijing: Beijing University of Technology: 7-38
卢俊达. 2017. 基于深度学习的电力设备故障诊断专家系统. 北京: 北京工业大学: 7-38
Ma L, Ming Y, Guo T, Liao S, Zou Y X and Xiong Y. 2020. Method for anti-destruction early warning system of electric equipment based on video monitoring. Information Technology, 44(4): 115-120
马莉, 明月, 郭婷, 廖爽, 邹雨馨, 熊一. 2020. 基于视频监控的电力设备防破坏预警系统的方法. 信息技术, 44(4): 115-120 [DOI:10.13274/j.cnki.hdzj.2020.04.025]
Mikolov T, Chen K, Corrado G and Dean J. 2013. Efficient estimation of word representations in vector space[EB/OL]. [2020-07-22] . https://arxiv.org/pdf/1301.3781.pdf https://arxiv.org/pdf/1301.3781.pdf
Redmon J and Farhadi A. 2017. YOLO9000: better,faster, stronger//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 6517-6525[ DOI: 10.1109/CVPR.2017.690 http://dx.doi.org/10.1109/CVPR.2017.690 ]
Ren S Q, He K M, Girshick R and Sun J. 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 1137-1149[DOI:10.1109/TPAMI.2016.2577031]
Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2020-07-22] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf
Tang K H, Zhang H W, Wu B Y, Luo W H and Liu W. 2019. Learning to compose dynamic tree structures for visual contexts//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 6612-6621[ DOI: 10.1109/CVPR.2019.00678 http://dx.doi.org/10.1109/CVPR.2019.00678 ]
Wang B X. 2019. Research on Power Vision Terminal Target Detection System Based on Deep Learning. Beijing: North China Electric Power University (Beijing): 20-37
王碧霄. 2019. 基于深度学习的电力视觉终端目标检测系统研究. 北京: 华北电力大学(北京): 20-37
Wang X L, Shrivastava A and Gupta A. 2017. A-Fast-RCNN: hard positive generation via adversary for object detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 3039-3048[ DOI: 10.1109/CVPR.2017.324 http://dx.doi.org/10.1109/CVPR.2017.324 ]
Xu D F, Zhu Y K, Choy C B and Li F F. 2017. Scene graph generation by iterative message passing//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 3097-3106[ DOI: 10.1109/CVPR.2017.330 http://dx.doi.org/10.1109/CVPR.2017.330 ]
Zellers R, Yatskar M, Thomson S and Choi Y. 2018. Neural motifs: scene graph parsing with global context//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 5831-5840[ DOI: 10.1109/CVPR.2018.00611 http://dx.doi.org/10.1109/CVPR.2018.00611 ]
Zeng X W and Ran Z W. 2015. Electric power tower stress comprehensive monitoring and risk pre-warning system. Guizhou Electric Power Technology, 18(8): 70-72
曾宪武, 冉祯伟. 2015. 电力杆塔应力综合监测危险预警系统. 贵州电力技术, 18(8): 70-72 [DOI:10.19317/j.cnki.1008-083x.2015.08.023]
Zhang H W, Kyaw Z, Chang S F and Chua T S. 2017. Visual translation embedding network for visual relation detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 3107-3115[ DOI: 10.1109/CVPR.2017.331 http://dx.doi.org/10.1109/CVPR.2017.331 ]
Zhou H, Zhang C Y and Hu C P. 2019. Visual relationship detection with relative location mining//Proceedings of the 27th ACM International Conference on Multimedia. Nice, France: Association for Computing Machinery: 30-38[ DOI: 10.1145/3343031.3351024 http://dx.doi.org/10.1145/3343031.3351024 ]
相关文章
相关作者
相关机构
京公网安备11010802024621
