Continuous graph convolutional model for video smoke detection

Longzhen Yang; Feiniu Yuan; Shouyuan Yang; Bangjun Lei; Xiangfen Zhang

doi:10.11834/jig.190232

Fire and Smoke Special Forum | Views : 0 下载量: 42 CSCD: 2

PDF
Export
Share
Collection
Album

Continuous graph convolutional model for video smoke detection
Vol. 24, Issue 10, Pages: 1658-1669(2019)
Received：22 May 2019，

Revised：2019-7-13，

Accepted：20 July 2019，

Published：16 October 2019
DOI： 10.11834/jig.190232
稿件说明：

移动端阅览

DOI：

Longzhen Yang, Feiniu Yuan, Shouyuan Yang, Bangjun Lei, Xiangfen Zhang. Continuous graph convolutional model for video smoke detection[J]. Journal of Image and Graphics, 2019, 24(10): 1658-1669. DOI： 10.11834/jig.190232.

摘要

目的

视频烟雾检测在火灾预警中起到重要作用，目前基于视频的烟雾检测方法主要利用结构化模型提取烟雾区域的静态和动态特征，在时间和空间上对烟雾信息作同等或相似处理，忽略了视频数据在时间线上的连续性和特征的非结构化关系。图卷积网络（GCN）与神经常微分方程（ODE）在非欧氏结构与连续模型处理上具有突出优势，因此将二者结合提出了一种基于视频流和连续时间域的图烟雾检测模型。

方法

目前主流的视频烟雾检测模型仍以离散模型为基础，以规则形式提取数据特征，利用ODE网络构建连续时间模型，捕捉视频帧间的隐藏信息，将原本固定时间跨度的视频帧作为连续时间轴上的样本点，充分利用模型的预测功能，补充帧间丢失信息并对未来帧进行一定程度的模拟预测，生成视频帧的特征并交给图卷积网络对其重新建模，最后使用全监督和弱监督两种方法对特征进行分类。

结果

分别在2个视频和4个图像数据集上进行训练与测试，并与最新的主流深度方法进行了比较，在KMU（Korea Maritime University）视频数据集中，相比于性能第2的模型，平均正样本正确率（ATPR值）提高了0.6%；在2个图像数据集中，相比于性能第2的模型，正确率分别提高了0.21%和0.06%，检测率分别提升了0.54%和0.28%，在视频单帧图像集上正确率高于第2名0.88%。同时也在Bilkent数据集中进行了对比实验，以验证连续隐态模型在烟雾动态和起烟点预测上的有效性，对比实验结果表明所提连续模型能够有效预测烟雾动态并推测烟雾起烟点位置。

结论

提出的连续图卷积模型，综合了结构化与非结构化模型的优势，能够获得烟雾动态信息，有效推测烟雾起烟点位置，使烟雾检测结果更加准确。

Abstract

Objective

Video smoke detection plays an important role in real-time fire alarms by solving the limitation of applying sensors in large spaces

outdoors

and other types of environment with strong air turbulence. Current video-based methods mainly extract the static and dynamic features of smoke and process them with the same structured model

which may disregard its continuous information and unstructured feature properties. Graph convolutional networks (GCNs) and neural ordinary differential equations (ODEs) exhibit powerful strength on processing non-Euclidean structures and continuous timeline models. Therefore

these methods can ultimately be utilized in video smoke detection. On account of the success of these new methods

we propose a flow-based continuous graph model for video smoke detection.

Method

In this study

we constructed a continuous timeline model using a neural ODE network

while most methods in the video smoke detection domain remain focused on discrete spatial-temporal features in the Euclidean space. We considered video frames with fixed time spans as sample points on a continuous timeline. By simulating the latent space of hidden variables through the latent time series model

we could obtain the hidden information between frames

which may be disregarded in discrete models. When the model was established

the lost between-frames information and the short-term future frame can be predicted. Through this procedure

we could effectively advance fire alarms. For the detection functions

we used GCNs to extract the feature of the video frame (or block)

which will be trained for classification by utilizing fully and weakly supervised methods. Considering the lack of smoke labels for bounding boxes or pixel-level ground truths in real smoke video datasets

we pretrained our model on a number of smoke images and used it to predict the label of sliding windows in a labeled video frame. This process was conducted to find the origin fire point or predict the motion information of smoke.

Result

We compare our model with seven state-of-the-art models of video smoke detection and five image-detection models

including the traditional approaches and deep-learning methods on two video and four image datasets. The video data are collected from KMU

Bilkent

USTC

and Yuan. The quantitative evaluation metrics contain detection rate (DR)

accuracy rate (AR)

false alarm rate

average true positive rate (ATPR)

average false positive rate

average true negative rate

and F-measure (F2). We provide several latent models of each method for comparisons. Experimental results show that our model outperforms most of other methods in KMU and the Yuan datasets. The visualized detection samples show that our model can capture the dynamic motion feature of smoke and predict the origin fire point by combining these features. Comparative experiments demonstrate that the continuous model improves smoke detection accuracy. Compared with the 3D parallel convolutional network and other results in the KMU video dataset

ATPRs increase by 0.6%. Compared with DMCNN and other results in the Yuan image datasets

obtained ARs increase by 0.21% and 0.06% on image datasets

respectively. Although the results of the state-of-art models are over 98%

we also achieve DR increases by 0.54% and 0.28%. In addition

we conduct a series of experiments in the Bilkent video datasets to verify the effectiveness and robustness of our latent model on the prediction of smoke motion. As shown in the separated screenshots of the real smoke videos

we initially sample several frames randomly and slide the bounding box window to divide the image block and predict their labels using our continuous graph convolutional model. We use the pretrained model given that the real smoke videos do not have specific labels for bounding boxes or pixels. Thereafter

we feed the center point of these samples to our latent model and predict the labels of the bounding boxes in the current image. Through visualizing the smoke areas detected by our model

we find that our latent model correctly tracks the diffusion direction of smoke and updates its locations. By reversing the timeline fed to the latent model

we can obtain the trajectory of smoke fusion back to its origin point. Therefore

the effectiveness of our latent model to predict smoke motion and infer the origin fire point is demonstrated. However

a quantized verification has not been conducted yet.

Conclusion

In this study

we propose a video-based continuous graph convolutional model that combines the strength of structured and unstructured models. We also capture the dynamic information of smoke and effectively predict the origin fire point. Experiment results show that our model outperforms several state-of-the-art approaches of video and image smoke detection.

关键词

Keywords

references

Yuan F N. Motion accumulation and translucence based model for video smoke detection[J]. Journal of Data Acquisition&Processing, 2007, 22(4):396-400.

袁非牛.基于运动累积和半透明的视频烟雾探测模型[J].数据采集与处理, 2007, 22(4):396-400[DOI:10.3969/j.issn.1004-9037.2007.04.002]

Yuan F N. A fast accumulative motion orientation model based on integral image for video smoke detection[J]. Pattern Recognition Letters, 2008, 29(7):925-932.[DOI:10.1016/j.patrec.2008.01.013]

Yuan F N, Zhang Y M, Liu S X, et al. Video smoke detection based on accumulation and main motion orientation[J]. Journal of Image and Graphics, 2008, 13(4):808-813.

袁非牛, 张永明, 刘士兴, 等.基于累积量和主运动方向的视频烟雾检测方法[J].中国图象图形学报, 2008, 13(4):808-813.[DOI:10.11834/jig.20080434]

Zhao Y Q, Zhou Z, Xu M M. Forest fire smoke video detection using spatiotemporal and dynamic texture features[J]. Journal of Electrical and Computer Engineering, 2015, 2015:706187.[DOI:10.1155/2015/706187]

Zhou Z Q, Shi Y S, Gao Z F, et al. Wildfire smoke detection based on local extremal region segmentation and surveillance[J]. Fire Safety Journal, 2016, 85:50-58.[DOI:10.1016/j.firesaf.2016.08.004]

Appana D K, Islam R, Khan S A, et al. A video-based smoke detection using smoke flow pattern and spatial-temporal energy analyses for alarm systems[J]. Information Sciences, 2017, 418-419:91-101.[DOI:10.1016/j.ins.2017.08.001]

Frizzi S, Kaabi R, Bouchouicha M, et al. Convolutional neural network for video fire and smoke detection[C]//Proceedings of the 42nd Annual Conference of the IEEE Industrial Electronics Society. Florence, Italy: IEEE, 2016: 877-882.[ DOI: 10.1109/IECON.2016.7793196 http://dx.doi.org/10.1109/IECON.2016.7793196 ]

Pundir A S, Raman B. Deep belief network for smoke detection[J]. Fire Technology, 2017, 53(6):1943-1960.[DOI:10.1007/s10694-017-0665-z]

Hu Y C, Lu X B. Real-time video fire smoke detection by utilizing spatial-temporal ConvNet features[J]. Multimedia Tools and Applications, 2018, 77(22):29283-29301.[DOI:10.1007/s11042-018-5978-5]

Li X Q, Chen Z X, Wu Q M J, et al. 3D parallel fully convolutional networks for real-time video wildfire smoke detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018.[DOI:10.1109/TCSVT.2018.2889193]

Lin G H, Zhang Y M, Xu G, et al. Smoke detection on video sequences using 3D convolutional neural networks[J]. Fire Technology, 2019:1-21.[DOI:10.1007/s10694-019-00832-w]

Lin G H, Zhang Y M, Zhang Q X, et al. Smoke detection in video sequences based on dynamic texture using volume local binary patterns[J]. Ksii Transactions on Internet and Information Systems, 2017, 11(11):5522-5536.[DOI:10.3837/tiis.2017.11.019]

Paygude S, Vyas V. Dynamic texture segmentation approaches for natural and manmade cases:survey and experimentation[J]. Archives of Computational Methods in Engineering, 2018:1-13.[DOI:10.1007/s11831-018-09305-9]

Yuan F N, Zhang L, Wan B Y, et al. Convolutional neural networks based on multi-scale additive merging layers for visual smoke recognition[J]. Machine Vision and Applications, 2019, 30(2):345-358.[DOI:10.1007/s00138-018-0990-3]

Zhang Q X, Lin G H, Zhang Y M, et al. Wildland forest fire smoke detection based on faster R-CNN using synthetic smoke images[J]. Procedia Engineering, 2018, 211:441-446.[DOI:10.1016/j.proeng.2017.12.034]

Yin M X, Lang C Y, Li Z, et al. Recurrent convolutional network for video-based smoke detection[J]. Multimedia Tools and Applications, 2019, 78(1):237-256.[DOI:10.1007/s11042-017-5561-5]

Bruna J, Zaremba W, Szlam A, et al. Spectral networks and iocally connected networks on graphs[EB/OL].[2019-05-07] . https://arxiv.org/pdf/1312.6203v3.pdf https://arxiv.org/pdf/1312.6203v3.pdf .

Duvenaud D K, Maclaurin D, Aguileraiparraguirre J, et al. Convolutional networks on graphs for learning molecular fingerprints[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems(NeurIPS), Montréal, Canada: MIT Press, 2015: 2224-2232.

Li Y T, Tarlow D, Brockschmidt M, et al. Gated graph sequence neural networks[EB/OL].[2019-05-07] . https://arxiv.org/pdf/1511.05493.pdf https://arxiv.org/pdf/1511.05493.pdf .

Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[EB/OL].[2019-05-07] . https://arxiv.ovg/pdf/1609.02907.pdf https://arxiv.ovg/pdf/1609.02907.pdf .

Chen T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations[EB/OL].[2019-05-07] . https://arxiv.org/pdf/1806.07366.pdf https://arxiv.org/pdf/1806.07366.pdf .

Toreyin B U, Cetin A E. Wildfire detection using LMS based active learning[C]//Proceedings of the 2009 IEEE International Conference on Acoustics, Speech, and Signal Processing, Taipei, China: IEEE, 2009: 1461-1464.[ DOI: 10.1109/ICASSP.2009.4959870 http://dx.doi.org/10.1109/ICASSP.2009.4959870 ]

Ham S, Ko B C, Nam J Y. Vision based forest smoke detection using analyzing of temporal patterns of smoke and their probability models[C]//Proceeedings of SPIE 7877, Image Processing: Machine Vision Applications Ⅳ. San Francisco, California: SPIE, 2011.[ DOI: 10.1117/12.871995 http://dx.doi.org/10.1117/12.871995 ]

Ko B C, Kwak J Y, Nam J Y. Wildfire smoke detection using temporospatial features and random forest classifiers[J]. Optical Engineering, 2012, 51(1):017208.[DOI:10.1117/1.OE.51.1.017208]

Park J, Ko B, Nam J Y, et al. Wildfire smoke detection using spatiotemporal bag-of-features of smoke[C]//Proceedings of the Workshop on Applications of Computer Vision. Tampa, FL, USA: IEEE, 2013: 200-205.[ DOI: 10.1109/WACV.2013.6475019 http://dx.doi.org/10.1109/WACV.2013.6475019 ]

Yin Z J, Wan B Y, Yuan F N, et al. A deep normalization and convolutional neural network for image smoke detection[J]. IEEE Access, 2017, 5:18429-18438.[DOI:10.1109/ACCESS.2017.2747399]

Alert me when the article has been cited

提交

Video smoke detection: a literature survey

Bifurcated attention and feature interaction for few-shot fine-grained learning

Survey of anomaly detection methods in surveillance videos based on deep learning

Few-shot SAR image classification： a survey

pFedWSD： unified weakly supervised personalized federated image segmentation via similarity-aware distillation