发布时间: 2018-09-16 摘要点击次数: 全文下载次数: DOI: 10.11834/jig.170633 2018 | Volume 23 | Number 9 图像分析和识别

1. 首都师范大学北京成像技术高精尖创新中心, 北京 100048;
2. 首都师范大学资源环境与旅游学院, 北京 100048;
3. 首都师范大学三维数据获取与应用重点实验室, 北京 100048
 收稿日期: 2017-12-12; 修回日期: 2018-03-29 基金项目: 国家自然科学基金项目（41371434） 第一作者简介: 王志旋, 1992年生, 女, 硕士研究生, 研究方向为3维信息获取与应用。E-mail:wzx0815xxx@163.com;谢东海, 男, 副教授, 研究方向为摄影测量与计算机视觉。E-mail:xdhbj@126.com. 中图法分类号: P237 文献标识码: A 文章编号: 1006-8961(2018)09-1371-11

# 关键词

Faster R-CNN; 深度学习; 路灯检测; 全景; 前方交会; 核线约束

Automatically measuring the coordinates of streetlights in vehicle-borne spherical images
Wang Zhixuan, Zhong Ruofei, Xie Donghai
1. Beijing Advanced Innovation Center for Imaging Technology, Capital Normal University, Beijing 100048, China;
2. College of Resources Environment and Tourism, Capital Normal University, Beijing 100048, China;
3. Key Lab of 3D Information Acquisition and Application, Capital Normal University, Beijing 100048, China
Supported by: National Natural Science Foundation of China (41371434)

# Abstract

Objective With the development of urban management, a growing number of cities are implementing coding projects for streetlight poles. In such projects, the coordinates of streetlamps are obtained and serial numbers are assigned to them. The coordinates can be obtained in many ways, such as RTK and laser measurements. A quick and easy approach to obtain the data is required because tens of thousands of streetlamps are present in a city. In consideration of the cost, mobile panorama measurement is preferred. However, most current panorama measurements are conducted by means of human-computer interaction, in which homologous image points are selected to perform forward intersection to obtain the coordinates. This approach consumes substantial energy and time. Therefore, in this paper, we propose an automatic method to obtain the coordinates of streetlamps by combining object detection with panoramic measurement. Method The method combines deep learning and panoramic measurement to automatically obtain the coordinates of streetlight poles. No feature points are obvious on the poles because of their rod-shaped features, and the top of the streetlamp is different because of the different design. The distortion of panoramic images strongly influences the detection of the top of a streetlamp. Thus, the bottom of the poles is used as the detection target in this paper. The pole bottoms are detected by faster R-CNN. Meanwhile, the coordinate file that contains the upper-left and lower-right corners of the detection frames are output and compared with the detection results obtained by the combination of histogram of oriented gradient (HOG) and support vector machine (SVM). Then, the diagonal intersection of the detection box is regarded as the foot of the streetlight pole, and an epipolar line is used to find homologous image points in two panoramic images because multiple streetlight poles can be present in a panoramic image. Based on the matching results, the space coordinates of the streetlight poles are obtained by forward intersection of the panoramas, thereby confirming the potential of this preliminary work for the coding projects. Result The aforementioned two methods were used to detect the streetlights of 100 panoramic images, which include 162 streetlights. A total of 1826 detection results were obtained based on HOG features, of which the correct bottom of streetlamps is 142. A total of 149 detection results are based on the faster R-CNN, of which 137 are correct. We can conclude that the faster R-CNN has obvious advantages. Thus, in this study, we use the faster R-CNN combined with panoramic measurement to automatically obtain the streetlight coordinates. The distance from the bottom of the streetlamp to the two imaging centers and the intersection angle formed by the three points significantly affect the accuracy of coordinate measurement. To filter out the coordinates that are less affected by the aforementioned two factors, we compare measurement results, which are the distances of approximately 7, 11, and 18m; the intersection angles are 0° to 180°. We have verified that when the intersection angles are from 30° to 150°, the influence on the measurement accuracy is smaller because the distance is closer. Based on the preceding rules, 120 coordinates of streetlamps are selected to determine the statistical distribution of the intersection angle and distance. Points with a distance of less than 20 m and intersection angle greater than 30° and less than 150° are selected for the coordinate error analysis, and 102 points meet the requirements for accuracy verification. The deviation of space coordinate measurement is less than 0.3 m and the maximum is not more than 0.6 m, thereby satisfying the requirement that the accuracy of the coordinates is within 1 m. Conclusion This paper presents a method of automatically obtaining the coordinates of streetlamps. The method of target detection based on deep learning is applied to the panorama measurement. The method avoids the manual selection of the homologous image points for measurement, which saves considerable labor and material resources. We conclude that this method exhibits practical significance because it is suitable for road sections or periods with low traffic volume in the city, thereby preventing excessive obstruction caused by vehicles. However, for panoramas with seriously obstructed streetlights, this method has certain limitations.

# Key words

faster region convolutional neural network (Faster R-CNN); deep learning; street light pole detection; panorama; forward intersection; epipolar geometry

# 2.3 核线约束

 $\mathit{\boldsymbol{N}} = {(\mathit{\boldsymbol{E}}{\mathit{\boldsymbol{P}}_0})^{\rm{T}}}$ (1)

 $L = r({\rm{ \mathsf{ π} }}/2 - \beta )$ (2)

# 2.4 全景图像前方交会

 $\left\{ \begin{array}{l} a = 2{\rm{ \mathsf{ π} }}r\\ \alpha = \frac{x}{r}\\ \theta = \frac{y}{r} \end{array} \right.$ (3)

 $\left\{ \begin{array}{l} X = r \cdot {\rm{sin}}\left( \alpha \right){\rm{sin}}\left( \theta \right)\\ Y = r \cdot {\rm{cos}}\left( \alpha \right){\rm{sin}}\left( \theta \right)\\ Z = r \cdot {\rm{cos}}\left( \alpha \right) \end{array} \right.$ (4)

 $\left[ \begin{array}{l} X\\ Y\\ Z \end{array} \right] \approx \mathit{\boldsymbol{R}}\left[ \begin{array}{l} {X_{\rm{g}}}\\ {Y_{\rm{g}}}\\ {Z_{\rm{g}}} \end{array} \right] + \mathit{\boldsymbol{T}}$ (5)

 $\mathit{\boldsymbol{R}} = \left[ {\begin{array}{*{20}{c}} {{r_0}}&{{r_1}}&{{r_2}}\\ {{r_3}}&{{r_4}}&{{r_5}}\\ {{r_6}}&{{r_7}}&{{r_8}} \end{array}} \right],\mathit{\boldsymbol{T}} = \left[ \begin{array}{l} {t_0}\\ {t_1}\\ {t_2} \end{array} \right]$ (6)

 $\left\{ \begin{array}{l} \frac{X}{Z} = \frac{{{r_0}{X_{\rm{g}}} + {r_1}{Y_{\rm{g}}} + {r_2}{Z_{\rm{g}}} + {t_0}}}{{{r_6}{X_{\rm{g}}} + {r_7}{Y_{\rm{g}}} + {r_8}{Z_{\rm{g}}} + {t_2}}}\\ \frac{Y}{Z} = \frac{{{r_3}{X_{\rm{g}}} + {r_4}{Y_{\rm{g}}} + {r_5}{Z_{\rm{g}}} + {t_1}}}{{{r_6}{X_{\rm{g}}} + {r_7}{Y_{\rm{g}}} + {r_8}{Z_{\rm{g}}} + {t_2}}} \end{array} \right.$ (7)

 ${x_{\rm{n}}} = \frac{X}{Z},{\rm{ }}\;\;\;{y_{\rm{n}}} = \frac{Y}{Z}$ (8)

 $\begin{array}{l} ({r_0} - {x_{\rm{n}}}{r_6}){X_{\rm{g}}} + ({r_1} - {x_{\rm{n}}}{r_7}){Y_{\rm{g}}} + \\ \;\;\;\;({r_2} - {x_{\rm{n}}}{r_8}){Z_{\rm{g}}} = {x_{\rm{n}}}{t_2} - {t_0} \end{array}$ (9)

 $\begin{array}{l} ({r_3} - {y_{\rm{n}}}{r_6}){X_{\rm{g}}} + ({r_4} - {y_{\rm{n}}}{r_7}){Y_{\rm{g}}} + \\ \;\;\;\;\;({r_5} - {y_{\rm{n}}}{r_8}){Z_{\rm{g}}} = {y_{\rm{n}}}{t_2} - {t_1} \end{array}$ (10)

# 3 实验结果与分析

Table 1 Comparison of test results

 检测方法 检测总数/个 正确目标/个 实际目标/个 准确率/% 完整度/% HOG+SVM 1 826 142 162 7.78 87.65 Faster R-CNN 149 137 162 91.95 84.57

Table 2 RMSE of coordinates

 X/m Y/m Z/m 中误差 0.201 0.202 0.102 最大误差 0.555 0.538 0.445 最小误差 0.004 0.002 0.001

# 参考文献

• [1] Guo L, Yang Y C, Li B J. Location method of urban alarm based on street lamp[J]. Urban Geotechnical Investigation & Surveying, 2009(3): 35–38. [郭岚, 杨永崇, 历保军. 基于路灯的城市110报警定位方法的研究[J]. 城市勘测, 2009(3): 35–38. ] [DOI:10.3969/j.issn.1672-8262.2009.03.011]
• [2] Yu Y T, Li J, Guan H Y, et al. Semiautomated extraction of street light poles from mobile LiDAR point-clouds[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(3): 1374–1386. [DOI:10.1109/TGRS.2014.2338915]
• [3] Hu Y J, Li X, Xie J, et al. A novel approach to extracting lamps from vehicle-borne laser data[C]//Proceedings of the 19th International Conference on Geoinformatics. Shanghai, China: IEEE, 2011: 1-6. [DOI: 10.1109/GeoInformatics.2011.5981183]
• [4] Lehtomäki M, Jaakkola A, Hyyppä J, et al. Detection of vertical pole-like objects in a road environment using vehicle-based laser scanning data[J]. Remote Sensing, 2010, 2(3): 641–664. [DOI:10.3390/rs2030641]
• [5] Zheng H, Tan F T, Wang R S. Pole-Like object extraction from mobile lidar data[C]//International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, XLI-B1: 729-734. [DOI: 10.5194/isprs-archives-XLI-B1-729-2016]
• [6] Jiang Z D, Jiang N, Wang Y J, et al. Distance measurement in panorama[C]//Proceedings of 2007 IEEE International Conference on Image Processing. San Antonio, Texas: IEEE, 2007: Ⅵ-393-Ⅵ-396. [DOI: 10.1109/ICIP.2007.4379604]
• [7] Fangi G. Multiscale multiresolution spherical photogrammetry with long focal lenses for architectural surveys[C]//Proceedings of International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences. Newcastle upon Tyne: Commission V Symposium, 2010: 1-6.
• [8] Zeng F Y, Zhong R F, Song Y, et al. Vehicle panoramic image matching based on epipolar geometry and space forward intersection[J]. Journal of Remote Sensing, 2014, 18(6): 1230–1236. [曾凡洋, 钟若飞, 宋杨, 等. 车载全景影像核线匹配和空间前方交会[J]. 遥感学报, 2014, 18(6): 1230–1236. ] [DOI:10.11834/jrs.20144025]
• [9] Sun Z X, Zhong R F. Measurement scheme for panoramic images[J]. Journal of Applied Sciences-Electronics and Information Engineering, 2015, 33(4): 399–406. [孙振兴, 钟若飞. 一种用于全景影像的测量方案[J]. 应用科学学报, 2015, 33(4): 399–406. ] [DOI:10.3969/j.issn.0255-8297.2015.04.006]
• [10] Kumar S, Deshpande A, Ho S S, et al. Urban street lighting infrastructure monitoring using a mobile sensor platform[J]. IEEE Sensors Journal, 2016, 16(12): 4981–4994. [DOI:10.1109/JSEN.2016.2552249]
• [11] Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio: IEEE, 2014: 580-587. [DOI: 10.1109/CVPR.2014.81]
• [12] Uijlings J R R, van de Sande K E A, Gevers T, et al. Selective search for object recognition[J]. International Journal of Computer Vision, 2013, 104(2): 154–171. [DOI:10.1007/s11263-013-0620-5]
• [13] He K M, Zhang X Y, Ren S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904–1916. [DOI:10.1109/TPAMI.2015.2389824]
• [14] Girshick R. Fast R-CNN[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 1440-1448. [DOI: 10.1109/ICCV.2015.169]
• [15] Ren S Q, He K M, Girshick R, et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137–1149. [DOI:10.1109/TPAMI.2016.2577031]
• [16] ZeilerM D, Fergus R. Visualizing and understanding convolutional networks[C]//Proceedings of the 13th European Conference on Computer Vision. Zurich: Springer, 2014: 818-833. [DOI: 10.1007/978-3-319-10590-1_53]
• [17] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10)[2017-11-30]. https://arxiv.org/abs/1409.1556.
• [18] Xie D H, Zhong R F, Wu Y, et al. Relative pose estimation and accuracy verification of spherical panoramic image[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(11): 1822–1829. [谢东海, 钟若飞, 吴俣, 等. 球面全景影像相对定向与精度验证[J]. 测绘学报, 2017, 46(11): 1822–1829. ] [DOI:10.11947/j.AGCS.2017.20160645]
• [19] Chen L W. Precision analysis of the plane-linear forward intersection in road surveying[J]. Ningxia Engineering Technology, 2010, 9(1): 87–89. [陈立文. 公路测量中平面前方交会点位的精度分析[J]. 宁夏工程技术, 2010, 9(1): 87–89. ] [DOI:10.3969/j.issn.1671-7244.2010.01.027]