Lightweight blurred car plate recognition method combined with generated images

Bin Duan; Xiang Fu; Yi Jiang; Jiexian Zeng

doi:10.11834/jig.190604

Image Analysis and Recognition | Views : 0 下载量: 22 CSCD: 2

PDF
Export
Share
Collection
Album

Lightweight blurred car plate recognition method combined with generated images
Vol. 25, Issue 9, Pages: 1813-1824(2020)
Received：21 November 2019，

Revised：2020-3-10，

Accepted：17 March 2020，

Published：16 September 2020
DOI： 10.11834/jig.190604
稿件说明：

移动端阅览

DOI：

Bin Duan, Xiang Fu, Yi Jiang, Jiexian Zeng. Lightweight blurred car plate recognition method combined with generated images[J]. Journal of Image and Graphics, 2020, 25(9): 1813-1824. DOI： 10.11834/jig.190604.

摘要

目的

模糊车牌识别是车牌识别领域的难题，针对模糊车牌图像收集困难、车牌识别算法模型太大、不适用于移动或嵌入式设备等不足，本文提出了一种轻量级的模糊车牌识别方法，使用深度卷积生成对抗网络生成模糊车牌图像，用于解决现实场景中模糊车牌难以收集的问题，在提升算法识别准确性的同时提升了部署泛化能力。

方法

该算法主要包含两部分，即基于优化卷积生成对抗网络的模糊车牌图像生成和基于深度可分离卷积网络与双向长短时记忆（long short-term memory，LSTM）的轻量级车牌识别。首先，使用Wasserstein距离优化卷积生成对抗网络的损失函数，提高生成车牌图像的多样性和稳定性；其次，在卷积循环神经网络的基础上，结合深度可分离卷积设计了一个轻量级的车牌识别模型，深度可分离卷积网络在减少识别算法计算量的同时，能对训练样本进行有效的特征学习，将特征图转换为特征序列后输入到双向LSTM网络中，进行序列学习与标注。

结果

实验表明，增加生成对抗网络生成的车牌图像，能有效提高本文算法、传统车牌识别和基于深度学习的车牌识别方法的识别率，为进一步提高各类算法的识别率提供了一种可行方案。结合深度可分离卷积的轻量级车牌识别模型，识别率与基于标准循环卷积神经网络（convolutional recurrent neural network，CRNN）的车牌识别方法经本文生成图像提高后的识别率相当，但在模型的大小和识别速度上都优于标准的CRNN模型，本文算法的模型大小为45 MB，识别速度为12.5帧/s，标准CRNN模型大小是82 MB，识别速度只有7帧/s。

结论

使用生成对抗网络生成图像，可有效解决模糊车牌图像样本不足的问题；结合深度可分离卷积的轻量级车牌识别模型，具有良好的识别准确性和较好的部署泛化能力。

Abstract

Objective

Blurred car license plate recognition is a bottleneck in the field of license recognition. The development of deep learning brings a new research direction for license recognition. Benefitting from the superfeature extraction power of convolutional neural network (CNN) and the good context learning capacity of convolutional recurrent neural network (CRNN)

the procedure of license recognition changes from a segmentation-based to an end-to-end method. Cur-rent deep learning-based license recognition methods suffer from two major problems. First

the size of the model is too large compared with traditional algorithms. In many applications

license recognition algorithms need to be deployed into embedded or mobile equipment. The generalization of algorithms is limited by its heavy size. Thus

keeping the balance between recognition efficiency and model size is challenging for license recognition methods based on deep learning. Second

the recognition effect of deep learning-based methods rely on large datasets

whereas the training dataset cannot be used widely because the car license formats vary in different countries. Collecting numerous car plate images manually is difficult in each country

let alone collecting blurred license images. Thus

the lack of training images is another challenge for deep learning-based license recognition methods. In this study

a lightweight car license plate recognition method is proposed. Car license plate images generated by optimized deep convolutional generative adversarial networks (DCGAN) are used to solve the problem of a lack of training dataset. The method simultaneously increases recognition accuracy and improves the generality of the model.

Method

The method includes two parts

namely

blurred car license image generation based on optimized DCGAN and lightweight car license plate recognition based on the depth of separable convolution networks with bidirectional long short-term memory (LSTM). In the first part

Wasserstein distance is used to optimize DCGAN and improve the diversity and stability of generated blurred images. Usually

generative adversarial networks (GAN) are designed to generate images with high quality; it is seldom used to generate images with low quality. Car license images are difficult to collect for format diversity and privacy reasons. In this study

GAN is used to generate images with low quality

that is

blurred car license images

to fill the training dataset and train deep models of license recognition. In the second part

a lightweight license recognition model is designed on the basis of deep separable convolution. Based on CRNN

the deep separable convolution operation can not only reduce the computation of recognition algorithms but also learn the features of training samples effectively. Lastly

the feature graphs are converted into feature sequences and input into bidirectional LSTM for sequence learning and labeling.

Result

Experimental results show that the proposed method is effective. First

by adding the generated blurred license images of DCGAN into the training dataset

recognition accuracy can be improved effectively

not only for the proposed method but also for the traditional method and other deep learning-based methods. Using generated image is a feasible scheme for all kinds of methods to improve their recognition accuracy. Second

by combining deep separable convolution layers with CRNN

the size of the recognition model is reduced

and the computation speed is improved effectively; furthermore

recognition accuracy is guaranteed. The proposed lightweight blurred car plate recognition model has a similar recognition accuracy with CRNN-based methods after improving its performance by using generated images in this research. By contrast

the size and recognition speed of the model are better than those of the CRNN model. For the proposed method

the size and recognition speed of the model are 45 MB and 12.5 frame/s

respectively; for the CRNN-based method

the values are 82 MB and 7 frame/s

respectively. The high recognition accuracy and small size of the model improve its possibility for application into mobile or embedded equipment.

Conclusion

The problem of lack blurred license images can be solved by using generated GAN images. Moreover

the lightweight license recognition model

which is improved with deep separable convolution

has high recognition accuracy and generalization performance.

关键词

Keywords

references

Anagnostopoulos C N E, Anagnostopoulos I E, Loumos V and Kayafas E. 2006. A license plate-recognition algorithm for intelligent transportation system applications. IEEE Transactions on Intelligent Transportation Systems, 7(3):377-392[DOI:10.1109/TITS.2006.880641]

Arjovsky M, Chintala S and Bottou L. 2017. Wasserstein GAN[EB/OL].[2019-11-01] . https://arxiv.org/pdf/1701.07875.pdf https://arxiv.org/pdf/1701.07875.pdf

Goel S and Dabas S. 2013. Vehicle registration plate recognition system using template matching//Proceedings of 2013 International Conference on Signal Processing and Communication. Noida: IEEE: 315-318[ DOI:10.1109/ICSPCom.2013.6719804 http://dx.doi.org/10.1109/ICSPCom.2013.6719804 ]

Goodfellow I I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th InternationalConference on Neural Information Processing Systems. Montreal: ACM: 2672-2680

Gou C, Wang K F, Yao Y J and Li Z X. 2016. Vehicle license plate recognition based on extremal regions and restricted Boltzmann machines. IEEE Transactions on Intelligent Transportation Systems, 17(4):1096-1107[DOI:10.1109/TITS.2015.2496545]

Graves A, Fernandez S, Gomez F and Schmidhuber J. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks//Proceedings of the 23rd International Conference on Machine Learning. Pittsburgh: ACM: 369-376[ DOI:10.1145/1143844.1143891 http://dx.doi.org/10.1145/1143844.1143891 ]

Guo J M and Liu Y F. 2008. License plate localization and character segmentation with feedback self-learning and hybrid binarization techniques. IEEE Transactions on Vehicular Technology, 57(3):1417-1424[DOI:10.1109/TVT.2007.909284]

Guo T Y, Wang B, Liu Y and Wei Y. 2019. Multi-channel fusion separable convolution neural networks for brain magnetic resonance image segmentation. Journal of Image and Graphics, 24(11):2009-2020

郭彤宇, 王博, 刘悦, 魏颖. 2019.多通道融合可分离卷积神经网络下的脑部磁共振图像分割.中国图象图形学报, 24(11):2009-2020)[DOI:10.11834/jig.190043]

Howard A, Zhu M L, Chen B, Kalenichenko D, Wang W J, Weyand T, Andreetto M and Adam H. 2017. MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL].[2019-11-01] . https://arxiv.org/pdf/1704.04861.pdf https://arxiv.org/pdf/1704.04861.pdf

Huang X, Wang M J and Gong M L. 2019. Hierarchically-fused generative adversarial network for text to realistic image synthesis//Proceedings of the 16th Conference on Computer and Robot Vision. Kingston: IEEE: 73-80[ DOI:10.1109/crv.2019.00018 http://dx.doi.org/10.1109/crv.2019.00018 ]

Isola P, Zhu J Y, Zhou T H and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 1125-1134[ DOI:10.1109/cvpr.2017.632 http://dx.doi.org/10.1109/cvpr.2017.632 ]

Jiao J B, Ye Q X and Huang Q M. 2009. A configurable method for multi-style license plate recognition. Pattern Recognition, 42(3):358-369[DOI:10.1016/j.patcog.2008.08.016]

Li H, Wang P, You M Y and Shen C H. 2018. Reading car license plates using deep neural networks. Image and Vision Computing, 72:14-23[DOI:10.1016/j.imavis.2018.02.002]

Llorens D, Marzal A, Palazón V and Vilar J M. 2005. Car license plates extraction and recognition based on connected components analysis and HMM decoding//Proceedings of the 2nd Iberian Conference on Pattern Recognition and Image Analysis. Estoril: Springer: 571-578[ DOI:10.1007/11492429_69 http://dx.doi.org/10.1007/11492429_69 ]

Luo X H, Wang P, Xiao H C, Li S and Sun D D. 2017. License plate recognition technology based on improved hidden Markov features. Foreign Electronic Measurement Technology,036(009):99-103

罗栩豪, 王培, 肖怀成, 李帅, 孙冬冬. 2017.基于改进隐马尔可夫特征的车牌识别技术.国外电子测量技术, 036(009):99-103)[DOI:10.3969/j.issn.1002-8978.2017.09.023]

Mehta R and Sivaswamy J. 2017. M-net: a convolutional neural network for deep brain structure segmentation//Proceedings of the 14th International Symposium on Biomedical Imaging. Melbourne: IEEE: 437-440[ DOI:10.1109/ISBI.2017.7950555 http://dx.doi.org/10.1109/ISBI.2017.7950555 ]

Nomura S, Yamanaka K, Katai O, Kawakami H and Shiose T. 2005. A novel adaptive morphological approach for degraded character image segmentation. Pattern Recognition, 38(11):1961-1975[DOI:10.1016/j.patcog.2005.01.026]

Pathak D, Krahenbühl P, Donahue J, Darrell T and Efros A A. 2016. Context encoders: feature learning by inpainting//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 2536-2544[ DOI:10.1109/CVPR.2016.278 http://dx.doi.org/10.1109/CVPR.2016.278 ]

Radford A, Metz L and Chintala S. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL]. 2016-12-20[2019-11-01] . https://arxiv.org/pdf/1511.06434.pdf https://arxiv.org/pdf/1511.06434.pdf

Ran L F. 2012. An algorithm of characters segmentation based on vertical projection for license plate. Communications Technology, 45(4):89-91, 98

冉令峰. 2012.基于垂直投影的车牌字符分割方法.通信技术, 45(4):89-91, 98)[DOI:10.3969/j.issn.1002-0802.2012.04.029]

Shi B G, Bai X and Yao C. 2017. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11):2298-2304[DOI:10.1109/TPAMI.2016.2646371]

Tao D P, Lin X, Jin L W and Li X L. 2016. Principal component 2-D long short-term memory for font recognition on single Chinese characters. IEEE Transactions on Cybernetics, 46(3):756-765[DOI:10.1109/TCYB.2015.2414920]

Wen Y, Lu Y, Yan J Q, Zhou Z Y, von Deneen K M and Shi P F. 2011. An algorithm for license plate recognition applied to intelligent transportation system. IEEE Transactions on Intelligent Transportation Systems, 12(3):830-845[DOI:10.1109/TITS.2011.2114346]

Yu Y, Tang B Y, Lin R L, Han S F, Tang T and Chen M. 2019. CWGAN: conditional Wasserstein generative adversarial nets for fault data generation//Proceedings of 2019 IEEE International Conference on Robotics and Biomimetics. Dali: IEEE: 2713-2718[ DOI:10.1109/robio49542.2019.8961501 http://dx.doi.org/10.1109/robio49542.2019.8961501 ]

Zhang C. 2016. An easy, flexible, and accurate plate recognition project for Chinese licenses in unconstrained situations[EB/OL].[2019-11-01] . https://github.com/zhang-can/EasyPR-DLL-CSharp https://github.com/zhang-can/EasyPR-DLL-CSharp

Alert me when the article has been cited

提交

The review of multi-focus image fusion methods based on deep learning

Enhanced attention-based joint semantic instance segmentation network for point clouds

Recent progress in rotation-invariant point cloud networks

Lightweight pyramid cross-attention network for orbital image defect detection

Dual-stage guided weakly supervised semantic segmentation with Gaussian correction