浏览全部资源
扫码关注微信
收稿：2020-03-20，
修回：2020-5-26，
录用：2020-6-3，
纸质出版：2021-03-16
移动端阅览
目的
2
随着智能交通领域车牌应用需求的升级，以及车牌图像复杂性的提高，自然场景下的车牌识别面临挑战。为应对自然场景下车牌的不规则变形问题，充分考虑车牌的形状特征，提出了一种自然场景下的变形车牌检测模型DLPD-Net（distorted license plate detection network）。
方法
2
该模型首次将免锚框目标检测方法应用于车牌检测任务中，不再使用锚框获取车牌候选区域，而是基于车牌热力值图与偏移值图来预测车牌中心；然后基于仿射变换寻找车牌角点位置，将变形车牌校正为接近于正面视角的平面矩形，从而实现在各种自然场景下变形车牌的检测。
结果
2
一方面，基于数据集CD-HARD评估DLPD-Net检测算法的性能；另一方面，基于数据集AOLP（the application-oriented license plate database）和CD-HARD评估基于DLPD-Net的车牌识别系统的有效性。实验结果表明，DLPD-Net具有更好的变形车牌检测性能，能够提升车牌识别系统的识别准确率，在数据集CD-HARD上识别准确率为79.4%，高出其他方法4.4% 12.1%，平均处理时间为237 ms。在数据集AOLP上取得了96.6%的识别准确率，未使用扩充数据集的情况下识别准确率达到了94.9%，高出其他方法1.6% 25.2%，平均处理时间为185 ms。
结论
2
本文提出的自然场景下的变形车牌检测模型DLPD-Net，能够实现在多种变形条件下的车牌检测，鲁棒性强，对遮挡、污垢和图像模糊等复杂自然环境下的车牌检测具有良好检测效果，同时，基于该检测模型的车牌识别系统在非受限的自然场景下具有更高的实用性。
Objective
2
As an important part of intelligent transportation systems
automatic license plate detection and recognition (ALPR) has always been a research hotspot in the field of computer vision. With the development of deep learning technology and new requirements for license plate recognition in the field of unmanned driving and safe cities as well as the upgrading challenges brought by complex license plate images taken by mobile phones and various mobile terminal devices
license plate recognition technology is now facing new challenges
mainly reflected in license plate background color; size and type varying in different countries; susceptibility of license plate images to complex environmental factors
such as poor lighting conditions
rain
snow
and complex background information interference; and diversity of acquisition equipment (such as mobile phone and law enforcement recorder) in real ALPR application
which leads to various irregular distortions of license plate images. The shape of a license plate is usually rectangular
with a fixed aspect ratio and definite color; hence
edge information and color features are frequently used to detect license plates in traditional ALPR techniques. These methods are highly efficient in controlled scenarios such as the entrance of a parking lot
but they are very sensitive to illumination variation
multiple viewpoints
stains
occlusion
image blur
and other influencing factors of the license plate image in natural scenarios
and the detection result is far from reaching application level. Methods based on deep learning technology have made remarkable achievements in license plate detection and character recognition tasks
and their recognition accuracy is higher than that of traditional ALPR techniques. However
they simply treat the license plate as a regular rectangular area and fail to consider the problem that the license plate will be distorted into an irregular quadrilateral in natural scenarios. These methods all use the anchor-based object detector to detect the license plate
but the size of the anchor is usually fixed
resulting in low detection accuracy for the object with a large distortion. License plates captured in natural scenarios are often distorted
especially in surveillance and cellphone videos; thus
the recognition accuracy of methods based on deep learning technology can still be improved. This paper designs a distorted license plate detection model in natural scenarios
named distorted license plate detection network (DLPD-Net)
to solve the problem of irregular
distorted license plate in natural scenarios and make full use of the license plate shape characteristics.
Method
2
For the first time
DLPD-Net applies the anchor-free object detection method to license plate detection. Instead of using the anchor to obtain the proposal license plate regions
it predicts the license plate center based on the heat map and offset map of the license plate. First
DLPD-Net uses ResNet-50 to extract the feature map of the input image
and then obtains the feature map of nine channels by using a detection block (including heat map
offset map
and affine transformation parameter map). Local peaks in the heat map are taken as the center of the license plate
and a square with fixed size is assumed at this location. Affine transformation parameters obtained by regression are used to construct the affine matrix
and the imaginary square is transformed into a quadrilateral corresponding to the shape of the license plate. Finally
the license plate region is obtained by using the offset value to translate the quadrilateral
then a distorted license plate is extracted and corrected to a plane rectangle similar to the front view. A complete loss function is designed
which consists of three parts
namely
heat map loss
offset loss
and affine loss
to train DLPD-Net effectively. Focal loss function is used to train the heat map and address the imbalance of positive and negative samples in license plate center prediction. L1 loss is used to train the offset map and obtain the local offset of each object center because the existence of the output stride will lead to the discretization error of real object coordinates. Affine loss is obtained by calculating the difference between the transformation value of the unit square's corners and the normalized value of the license plate's corners
and then summing.
Result
2
On the one hand
the performance of DLPD-Net is evaluated on the CD-HARD dataset
and results show that DLPD-Net could find the corners of distorted license plates well. On the other hand
based on DLPD-Net
this paper designs a distorted license plate recognition system in natural scenarios
which is composed of three modules: vehicle detection module
license plate detection
and correction module and license plate character recognition module. Experimental results show that compared with other commercial systems and license plate detection methods proposed in paper
DLPD-Net outperforms in distorted license plate detection and can improve the recognition accuracy of the license plate recognition system. In the CD-HARD dataset
the system's recognition accuracy is 79.4%
4.4%12.1% higher than that of other methods
and the average processing time is 237 ms. In the AOLP dataset
the system's recognition accuracy reaches 96.6%
and that is 94.9% without augmented samples
which is 1.6%25.2% higher than that of other methods
and the average processing time is 185 ms.
Conclusion
2
A distorted license plate detection model in natural scenarios
named DLPD-Net
is proposed. The model can extract the distorted license plate from the image and correct it into a plane rectangle similar to the front view
which is very useful for license plate character recognition. Based on DLPD-Net
an ALPR system is proposed. Experimental results show that DLPD-Net can achieve license plate detection under various distortion conditions in challenging datasets. It is robust and has a very good detection effect in complex natural scenarios such as occlusion
dirt
and image blur. The distorted license plate recognition system based on DLPD-Net is more practicable in unconstrained natural scenarios.
Bulan O, Kozitsky V, Ramesh P and Shreve M. 2017. Segmentation and annotation-free license plate recognition with deep localization and failure identification. IEEE Transactions on Intelligent Transportation Systems, 18(9): 2351-2363[DOI: 10.1109/TITS.2016.2639020]
Everingham M, Van Gool L, Williams C K I, Winn J and Zisserman A. 2010. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2): 303-338[DOI: 10.1007/s11263-009-0275-4]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Hsu G S, Ambikapathi A M, Chung S L and Su C P. 2017. Robust license plate detection in the wild//Proceedings of the 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Lecce, Italy: IEEE: 1-6[ DOI: 10.1109/AVSS.2017.8078493 http://dx.doi.org/10.1109/AVSS.2017.8078493 ]
Hsu G S, Chen J C and Chung Y Z. 2013. Application-oriented license plate recognition. IEEE Transactions on Vehicular Technology, 62(2): 552-561[DOI: 10.1109/TVT.2012.2226218]
Jaderberg M, Simonyan K, Zisserman A and Kavukcuoglu K. 2015. Spatial transformer networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge, Massachusetts, USA: MIT Press: 2017-2025
Laroca R, Severo E, Zanlorensi L A, Oliveira L S, Gonçalves G R, Schwartz W R and Menotti D. 2018. A robust real-time automatic license plate recognition based on the YOLO detector//Proceedings of 2018 International Joint Conference on Neural Networks (IJCNN). Rio de Janeiro, Brazil: IEEE: 1-10[ DOI: 10.1109/IJCNN.2018.8489629 http://dx.doi.org/10.1109/IJCNN.2018.8489629 ]
Law H and Deng J. 2020. CornerNet: detecting objects as paired keypoints. International Journal of Computer Vision, 128(3): 642-656[DOI: 10.1007/s11263-019-01204-1]
Li H and Shen C. 2016. Reading car license plates using deep convolutional neural networks and LSTMs[EB/OL]. 2016-01-21[2020-03-13] . http://arxiv.org/pdf/1601.05610.pdf http://arxiv.org/pdf/1601.05610.pdf
Lin T Y, Goyal P, Girshick R, He K M and Dollár P. 2017. Focal loss for dense object detection//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE: 2999-3007[ DOI: 10.1109/ICCV.2017.324 http://dx.doi.org/10.1109/ICCV.2017.324 ]
Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P and Zitnick C L. 2014. Microsoft COCO: common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 740-755[ DOI: 10.1007/978-3-319-10602-1_48 http://dx.doi.org/10.1007/978-3-319-10602-1_48 ]
Montazzolli and Jung C R. 2017. Real-time brazilian license plate detection and recognition using deep convolutional neural networks//Proceedings of the 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). Niteroi, Brazil: IEEE: 55-62[ DOI: 10.1109/SIBGRAPI.2017.14 http://dx.doi.org/10.1109/SIBGRAPI.2017.14 ]
Redmon J and Farhadi A. 2017. YOLO9000: better, faster, stronger//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE: 6517-6525[ DOI: 10.1109/CVPR.2017.690 http://dx.doi.org/10.1109/CVPR.2017.690 ]
Ren S Q, He K M, Girshick R and Sun J. 2015. Faster R-CNN: towards real-time object detection with region proposal networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge, Massachusetts, USA: MIT Press: 91-99
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z H, Karpathy A, Khosla A, Bernstein M, BergA C and Li F F. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3): 211-252[DOI: 10.1007/s11263-015-0816-y]
Silva S M and Jung C R. 2018. License plate detection and recognition in unconstrained scenarios//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 593-609[ DOI: 10.1007/978-3-030-01258-8_36 http://dx.doi.org/10.1007/978-3-030-01258-8_36 ]
Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition[EB/OL]. 2015-04-10[2020-03-13] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf
Tian J M, Wang G Y, Liu J G and Xia Y C. 2017. License plate detection in an open environment by density-based boundary clustering. Journal of Electronic Imaging, 26(3): #033017[DOI: 10.1117/1.JEI.26.3.033017]
Wang Y J, Pei M T and Jia Y D. 2014. License plate detection based on multiple features. Journal of Image and Graphics, 19(3): 471-475
王永杰, 裴明涛, 贾云得. 2014. 多信息融合的快速车牌定位. 中国图象图形学报, 19(3): 471-475[DOI: 10.11834/jig.20140318]
Wei T, Qiu S, Li C and Wang R. 2018. License plate detection algorithm based on computer multi scale assist. Acta Electronica Sinica, 46(9): 2188-2193
魏亭, 邱实, 李晨, 王锐. 2018. 计算机多尺度辅助定位车牌算法. 电子学报, 46(9): 2188-2193[DOI: 10.3969/j.issn.0372-2112.2018.09.020]
Xie L, Ahmad T, Jin L W, Liu Y L and Zhang S. 2018. A new CNN-based method for multi-directional car license plate detection. IEEE Transactions on Intelligent Transportation Systems, 19(2): 507-517[DOI: 10.1109/TITS.2017.2784093]
Ying T, Xin L and Li W X. 2018. License plate detection and localization in complex scenes based on deep learning//Proceedings of 2018 Chinese Control and Decision Conference (CCDC). Shenyang, China: IEEE: 6569-6574[ DOI: 10.1109/CCDC.2018.8408285 http://dx.doi.org/10.1109/CCDC.2018.8408285 ]
Zhou X Y, Wang D Q and Krähenbühl P. 2019. Objects as points[EB/OL].2019-04-16[2020-03-13] . https://arxiv.org/pdf/1904.07850.pdf https://arxiv.org/pdf/1904.07850.pdf
相关作者
相关机构
京公网安备11010802024621
