Survey of testing techniques of autonomous driving software

Yang Feng; Zhilong Xia; An Guo; Zhenyu Chen

doi:10.11834/jig.200493

Review | Views : 0 下载量: 712 CSCD: 7

PDF
Export
Share
Collection
Album

Survey of testing techniques of autonomous driving software
Vol. 26, Issue 1, Pages: 13-27(2021)
Received：20 August 2020，

Revised：2020-10-27，

Accepted：03 November 2020，

Published：16 January 2021
DOI： 10.11834/jig.200493
稿件说明：

移动端阅览

DOI：

Yang Feng, Zhilong Xia, An Guo, Zhenyu Chen. Survey of testing techniques of autonomous driving software[J]. Journal of Image and Graphics, 2021, 26(1): 13-27. DOI： 10.11834/jig.200493.

摘要

自动驾驶系统（autonomous driving system，ADS）是一种集成高精度传感器、人工智能和地图导航系统等模块的信息—物理融合系统。该类系统中的自动驾驶软件完成了从高级辅助驾驶到无人驾驶任务中关键的感知、定位、预测、规划和控制任务。随着深度学习和强化学习等人工智能技术的发展和车载硬件设备的不断升级，高级别的自动驾驶软件已经逐渐应用于多种安全攸关的场景中，保障其运行稳定性与可靠性的测试技术逐渐成为学术界和产业界的研究重点。本文在广泛调研国内外文献基础上，对自动驾驶软件测试技术进行了深入分析与梳理。结合自动驾驶软件的架构特点及系统特征，讨论了面向自动驾驶系统的仿真测试和实景测试，以及面向组件的测试技术。其中，在仿真方法方面，分析了软件仿真、半实体仿真和在环仿真等技术；在仿真对象方面，讨论了静态环境仿真、动态场景仿真、传感器仿真和车辆动力学仿真等。同时，本文介绍了当前实景测试的进展与情况，重点分析了实景测试案例中的得失优劣。在面向自动驾驶软件组件的测试技术方面，重点讨论了当前数据驱动技术在感知组件、决策规划组件，以及控制组件测试方面的进展。最后，本文总结分析了自动驾驶软件测试当前面临的挑战，并对未来自动驾驶软件测试技术的研究方向和研究重点进行了展望。

Abstract

An autonomous driving system(ADS) is a cyber-physical system that integrates a number of complicated software modules

such as high-precision sensors

artificial intelligence

and navigation systems. The autonomous driving software in this system performs sensing

positioning

forecasting

planning

and controlling tasks. With the development of artificial intelligence technologies and the continuous upgrading of onboard hardware devices

advanced autonomous driving software has been applied in a variety of safety-critical scenarios. Thus

the testing technology that assures its stability and reliability has naturally become the focus of academia and industry. To summarize the advance of the testing technology of autonomous driving software

this paper first characterizes the architecture and design of autonomous driving software. It presents the reference architecture of autonomous driving software and details the primary functionality of each component. It also introduces the interaction between these components and summarizes the features and challenges of autonomous driving software testing. Then

this paper extensively reviews the literature

providing a comprehensive discussion on the testing technology of autonomous driving software

which includes three related topics: simulation-based testing

real-scenario testing

and component-oriented testing. Simulation testing provides a method to examine the behaviors of real vehicle software in virtual environments. It constructs the internal and external factors and conditions that influence the software system to simulate various situations faced by autonomous vehicles with different degrees. This paper examines two critical components of simulation testing: the simulation methods and simulation targets. With regard to the simulation methods

software simulation

semi-physical simulation

and X-in-the-loop simulation are investigated. Currently

all these simulation methods are widely employed in autonomous driving software testing. They are capable of reflecting the behaviors of autonomous driving software under various virtual environments and thus enable engineers to test autonomous driving software at a much lower cost. For the simulation targets

this paper details the simulation of static environments

dynamic scenarios

sensors

and vehicle dynamics. Various simulation techniques are designed for these targets and are further employed to test different functionalities of autonomous driving software. For each simulation target

this paper discusses its usage and state-of-the-art simulation techniques. However

because simulation testing can reflect the behavior of autonomous driving software only in a virtual environment

it cannot completely represent the testing results under real scenarios. Compared with simulation testing

the cost of real scenario testing is relatively high

and the testing scale is often small. Thus

it cannot cover the input field and operating environment completely. However

real scenario testing is critical in quality assurance because it is the only way to identify the performance of autonomous driving software under real physical settings. This paper introduces the real-scenario testing cases conducted by the manufacturers of autonomous driving cars

which provide solid data that reflect the real road traffic scenario for the simulation testing of autonomous driving software. Component-oriented testing focuses on assuring the quality of individual components of autonomous driving software. In modern autonomous driving software

deep neural networks (DNNs) play a critical role. They are employed to assist in various driving tasks

such as perception

decision

and planning. However

testing the DNN-embedded software component is significantly different from conventional software components because their business logic is learned from massive data but is not defined with interpretable rules. To ensure the quality of these components

software engineers often adopt data-driven testing techniques. This paper introduces data-driven testing techniques for three primary components

i.e.

perception

decision and planning

and controlling. For the perception component

the primary challenge is to generate test data to input into various sensors

such as LiDAR

radar

and camera. The existing solutions are often built based on a mutation algorithm to augment the seed dataset

with the goal of increasing neuron coverages. For the decision and planning component

some researchers leverage reinforcement learning algorithms to combine traditional path planning algorithms

expert systems

and machine learning techniques to enhance the testing engines. For the controlling component

the testing often involves the hardware

including speed controller

steering controller

braking controller

and stability controller. Researchers have evaluated the performance and reliability of controller-oriented algorithms and explored automated test generation methods. Finally

this paper summarizes and analyzes the current challenges of automatic driving software testing and prospects for the future research direction and emphasis of automatic driving software testing technology.

关键词

Keywords

references

Ahamed M F S, Tewolde G and Kwon J. 2018. Software-in-the-loop modeling and simulation framework for autonomous vehicles//Proceedings of 2018 IEEE International Conference on Electro/Information Technology. Rochester, USA: IEEE: 305-310[ DOI: 10.1109/EIT.2018.8500101 http://dx.doi.org/10.1109/EIT.2018.8500101 ]

Behere S and Torngren M. 2015. A functional architecture for autonomous driving//Proceedings of the 1st International Workshop on Automotive Software Architecture. Montreal, Canada: IEEE: 3-10[ DOI: 10.1145/2752489.2752491 http://dx.doi.org/10.1145/2752489.2752491 ]

Bertozzi M, Bombini L, Broggi A, Buzzoni M, Cardarelli E, Cattani S, Cerri P, Coati A, Debattisti S, Falzoni A, Fedri ga R I, Felisa M, Gatti L, Giacomazzo A, Grisleri P, Laghi M C, Mazzei L, Medici P, Panciroli M, Porta P P, Zani P and Versari P. 2011. VIAC: an out of ordinary experiment//2011 IEEE Intelligent Vehicles Symposium. Baden-Baden, Germany: IEEE: 175-180[ DOI: 10.1109/IVS.2011.5940531 http://dx.doi.org/10.1109/IVS.2011.5940531 ]

Blanco M, Atwood J, Russell S, Trimble T, McClafferty J and Perez M. 2016. Automated Vehicle Crash Rate Comparison Using Naturalistic Data. Blacksburg, VA: Virginia Tech Transportation Institute

Briefs U. 2015. Mcity grand opening. Research Review, 46(3):1-10

Broggi A, Buzzoni M, Debattisti S, Grisleri P, Laghi M C, Medici P and Versari P. 2013. Extensive tests of autonomous driving technologies. IEEE Transactions on Intelligent Transportation Systems, 14(3):1403-1415[DOI:10.1109/TITS.2013.2262331]

Broggi A, Cerri P, Debattisti S, Laghi M C, Medici P, Panciroli M and Prioletti A. 2014. Proud-public road urban driverless test: architecture and results//2014 IEEE Intelligent Vehicles Symposium. Dearborn, USA: IEEE: 648-654[ DOI: 10.1109/IVS.2014.6856478 http://dx.doi.org/10.1109/IVS.2014.6856478 ]

Brogle C, Zhang C, Lim K L and Bräunl T. 2019. Hardware-in-the-loop autonomous driving simulation without real-time constraints. IEEE Transactions on Intelligent Vehicles, 4(3):375-384[DOI:10.1109/TIV.2019.2919457]

Buehler M, Iagnemma K and Singh S. 2009. The DARPA Urban Challenge:Autonomous Vehicles in City Traffic. Berlin, Heidelberg:Springer[DOI:10.1007/978-3-642-03991-1]

Canale M, Fagiano L and Razza V. 2010. Approximate NMPC for vehicle stability:design, implementation and SIL testing. Control Engineering Practice, 18(6):630-639[DOI:10.1016/j.conengprac.2010.03.002]

Chao Q, Bi H, Li W, Mao T, Wang Z, Lin MC and Deng Z. 2020. A survey on visual traffic simulation:models, evaluations, and applications in autonomous driving. Computer Graphics Forum, 39(1):287-308[DOI:10.1111/cgf.13803]

Chen Y, Chen S T, Zhang T, Zhang S Y and Zheng N N. 2018. Autonomous vehicle testing and validation platform: integrated simulation system with hardware in the loop//2018 IEEE Intelligent Vehicles Symposium. Changshu, China: IEEE: 949-956[ DOI: 10.1109/IVS.2018.8500461 http://dx.doi.org/10.1109/IVS.2018.8500461 ]

Chowdhury M, Gawande A and Wang L. 2017. Secure information sharing among autonomous vehicles in NDN//Proceedings of the 2nd IEEE/ACM International Conference on Internet-of-Things Design and Implementation. Pittsburgh, USA:IEEE: 15-26

Corso A, Du P, Driggs-Campbell K and Kochenderfer M J. 2019. Adaptive stress testing with reward augmentation for autonomous vehicle validatio//Proceedings of 2019 IEEE Intelligent Transportation Systems Conference. Auckland, New Zealand: IEEE: 163-168[ DOI: 10.1109/ITSC.2019.8917242 http://dx.doi.org/10.1109/ITSC.2019.8917242 ]

Debouk R I, Czerny B J, D'ambrosio J G, Joyce J, Osella M and Ramachandra V V. 2013. Method and system for ensuring operation of limited-ability autonomous driving vehicles. USA, No.8618922

Deng W W, Lee Y H and Zhao A N. 2008. Hardware-in-the-loop simulation for autonomous driving//Proceedings of the 34th Annual Conference of IEEE Industrial Electronics. Orlando, USA: IEEE: 1742-1747[ DOI: 10.1109/IECON.2008.4758217 http://dx.doi.org/10.1109/IECON.2008.4758217 ]

Duplouy Y. 2018. Applying Formal Methods to Autonomous Vehicle Control. Sacaly:Université Paris-Saclay

Emzivat Y, Ibañez-Guzmán J, Illy H, Martinet P and Roux O H. 2018. A formal approach for the design of a dependable perception system for autonomous vehicles//Proceedingsof the 21st International Conference on Intelligent Transportation Systems. Maui, USA: IEEE: 2452-2459[ DOI: 10.1109/ITSC.2018.8569903 http://dx.doi.org/10.1109/ITSC.2018.8569903 ]

Garcia J, Feng Y, Shen J J, Almanee S, Xia Y and Chen Q A. 2020. A comprehensive study of autonomous vehicle bugs//Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering. Seoul, South Korea: ACM: 385-396[ DOI: 10.1145/3377811.3380397 http://dx.doi.org/10.1145/3377811.3380397 ]

Geiger A, Lenz P, Stiller C and Urtasun R. 2013. Vision meets robotics:the KITTI dataset. The International Journal of Robotics Research, 32(11):1231-1237[DOI:10.1177/02783649134-91297]

Gietelink O, Ploeg J, De Schutter B and Verhaegen M. 2006. Development of advanced driver assistance systems with vehicle hardware-in-the-loop simulations. Vehicle System Dynamics, 44(7):569-590[DOI:10.1080/00423110600563338]

Gietelink O J, Verburg D J, Labibes K and Oostendorp A F. 2004. Pre-crash system validation with PRESCAN and VEHIL//2004 IEEE Intelligent Vehicles Symposium. Parma, Italy: IEEE: 913-918[ DOI: 10.1109/IVS.2004.1336507 http://dx.doi.org/10.1109/IVS.2004.1336507 ]

Helle P, Schamai W and Strobel C. 2016. Testing of autonomous systems-challenges and current state-of-the-art. INCOSE International Symposium, 26(1):571-584[DOI:10.1002/j.2334-5837.2016.00179.x]

Isele D, Rahimi R, Cosgun A, Subramanian K and Fujimura K. 2018. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning//Proceedings of 2018 IEEE International Conference on Robotics and Automation. Brisbane, Australia: IEEE: 2034-2039[ DOI: 10.1109/ICRA.2018.8461233 http://dx.doi.org/10.1109/ICRA.2018.8461233 ]

Jha S and Raman V. 2016. Automated synthesis of safe autonomous vehicle control under perception uncertainty//The 8th International Symposium on NASA Formal Methods. Minneapolis, USA: Springer: 117-132[ DOI: 10.1007/978-3-319-40648-0_10 http://dx.doi.org/10.1007/978-3-319-40648-0_10 ]

Jo K, Kim J, Kim D, Jang C and Sunwoo M. 2014. Development of autonomous car-Part I:distributed system architecture and development process. IEEE Transactions on Industrial Electronics, 61(12):7131-7140[DOI:10.1109/TIE.2014.2321342]

Jo K, Kim J, Kim D, Jang C and Sunwoo M. 2015. Development of autonomous car-Part Ⅱ:a case study on the implementation of an autonomous driving system based on distributed architecture. IEEE Transactions on Industrial Electronics, 62(8):5119-5132[DOI:10.1109/TIE.2015.2410258]

Joshi A. 2017. Powertrain and chassis hardware-in-the-loop (HIL) Simulation of Autonomous Vehicle Platform. No.2017-01-1991. SAE[ DOI: 10.4271/2017-01-1991 http://dx.doi.org/10.4271/2017-01-1991 ]

Jöckel L, Kläs M, and Martínez-Fernández S.2019.Safe traffic sign recognition through data augmentation for autonomous vehicles software//Proceedings of the 19th IEEE International Conference on Software Quality, Reliability and Security Companion.Sofia, Bulgaria: IEEE: 540-541[ DOI: 10.1109/QRS-C.2019.00114 http://dx.doi.org/10.1109/QRS-C.2019.00114 ]

Ko M K. 2015. Autonomous vehicle driving support system and autonomous driving method performed by the same. USA, No.14.132249

Koopman P and Wagner M. 2016. Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety, 4(1):15-24[DOI:10.4271/2016-01-0128]

Koopman P and Wagner M. 2017. Autonomous vehicle safety:an interdisciplinary challenge. IEEE Intelligent Transportation Systems Magazine, 9(1):90-96[DOI:10.1109/MITS.2016.2583491]

Krook J, Svensson L, Li Y C, Feng L and Fabian M. 2019. Design and formal verification of a safe stop supervisor for an automated vehicle//Proceedings of 2019 International Conference on Robotics and Automation. Montreal, Canada: IEEE: 5607-5613[ DOI: 10.1109/ICRA.2019.8793636 http://dx.doi.org/10.1109/ICRA.2019.8793636 ]

Kufieta K and Ditze M. 2017. A virtual environment for the development and validation of highly automated driving systems//The 17th Internationales Stuttgarter Symposium. Wiesbaden: Springer: 1391-1401[ DOI: 10.1007/978-3-658-16988-6_106 http://dx.doi.org/10.1007/978-3-658-16988-6_106 ]

Lee T B. 2019. Autopilot was active when a Tesla crashed into a truck, killing driver[EB/OL ] .[2020-03-30 ] . https://arstechnica.com/cars/2019/05/feds-autopilot-was-active-during-deadly-march-tesla-crash/ https://arstechnica.com/cars/2019/05/feds-autopilot-was-active-during-deadly-march-tesla-crash/

Levinson J, Askeland J, Becker J, Dolson J, Held D, Kammel S, Kolter J Z, Langer D, Pink O, Pratt V, Sokolsky M, Stanek G, Stavens D, Teichman A, Werling M and Thrun S. 2011. Towards fully autonomous driving : systems and algorithms//2011 IEEE Intelligent Vehicles Symposium. Baden-Baden, Germany: IEEE: 163-168[ DOI: 10.1109/IVS.2011.5940562 http://dx.doi.org/10.1109/IVS.2011.5940562 ]

Li L, Huang W L, Liu Y H, Zheng N N and Wang F Y. 2016a. Intelligence testing for autonomous vehicles:a new approach. IEEE Transactions on Intelligent Vehicles, 1(2):158-166[DOI:10.1109/TIV.2016.2608003]

Li N, Oyler D, Zhang M X, Yildiz Y, Girard A and Kolmanovsky I. 2016b. Hierarchical reasoning game theory based approach for evaluation and testing of autonomous vehicle control systems//Proceedings of the 55th IEEE Conference on Decision and Control. Las Vegas, USA: IEEE: 727-733[ DOI: 10.1109/CDC.2016.7798354 http://dx.doi.org/10.1109/CDC.2016.7798354 ]

Li N, Oyler D W, Zhang M X, Yildiz Y, Kolmanovsky I and Girard A R. 2018. Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems. IEEE Transactions on Control Systems Technology, 26(5):1782-1797[DOI:10.1109/TCST.2017.2723574]

Li W, Pan C W, Zhang R, Ren J P, Ma Y X, Fang J, Yan F L, Geng Q C, Huang X Y, Gong H J, Xu W W, Wang G P, Manocha D and Yang R G. 2019. AADS:augmented autonomous driving simulation using data-driven algorithms. Science Robotics, 4(28):#eaaw0863[DOI:10.1126/scirobotics.aaw0863]

Lin S C, Zhang Y Q, Hsu C H, Skach M, Haque E, Tang L J and Mars J. 2018. The architectural implications of autonomous driving: constraints and acceleration//Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems. Williamsburg, USA: ASPLOS: 751-766[ DOI: 10.1145/3173162.3173191 http://dx.doi.org/10.1145/3173162.3173191 ]

Lyu C, Cao D P, Zhao Y F, Auger D J, Sullman M, Wang H J, Dutka L M, Skrypchuk L and Mouzakitis A. 2018. Analysis of autopilot disengagements occurring during autonomous vehicle testing. IEEE/CAA Journal of Automatica Sinica, 5(1):58-68[DOI:10.1109/JAS.2017.7510745]

Maurer M, Gerdes J C, Lenz B and Winner H. 2016. Autonomous Driving:Technical, Legal and Social Aspects. Berlin, Heidelberg:Springer[DOI:10.1007/978-3-662-48847-8]

McCaffrey M. 2017. Unreal Engine VR Cookbook:Developing Virtual Reality with UE4. Boston:Addison-Wesley

McNaughton M, Urmson C, Dolan J M and Lee J W. 2011. Motion planning for autonomous driving with a conformal spatiotemporal lattice//Proceedings of 2011 IEEE International Conference on Robotics and Automation. Shanghai, China: IEEE: 4889-4 895[ DOI: 10.1109/ICRA.2011.5980223 http://dx.doi.org/10.1109/ICRA.2011.5980223 ]

Minnerup P and Knoll A. 2014. Testing autonomous driving systems against sensor and actuator error combinations//2014 IEEE Intelligent Vehicles Symposium. Dearborn, USA: IEEE: 561-566[ DOI: 10.1109/IVS.2014.6856565 http://dx.doi.org/10.1109/IVS.2014.6856565 ]

Navarro P J, Fernández C, Borraz R and Alonso D. 2017. A machine learning approach to pedestrian detection for autonomous vehicles using high-definition 3D range data. Sensors, 17(1):#18[DOI:10.3390/s17010018]

Papp Z, Labibes K, Thean A C and van Elk M G. 2003. Multi-agent based HIL simulator with high fidelity virtual sensors//IEEE IV2003 Intelligent Vehicles Symposium. Columbus, USA: IEEE: 213-218[ DOI: 10.1109/IVS.2003.1212911 http://dx.doi.org/10.1109/IVS.2003.1212911 ]

Pei K X, Cao Y Z, Yang J F and Jana S. 2017. DeepXplore: automated whitebox testing of deep learning systems//The 26th Symposium on Operating Systems Principles. Shanghai, China: ACM: 1-18[ DOI: 10.1145/3132747.3132785 http://dx.doi.org/10.1145/3132747.3132785 ]

Rakin A S, He Z Z and Fan D L. 2019. Bit-flip attack: CRUSHING neural network with progressive bit search//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, South Korea: IEEE: 1211-1220[ DOI: 10.1109/ICCV.2019.00130 http://dx.doi.org/10.1109/ICCV.2019.00130 ]

Rausch V, Hansen A, Solowjow E, Liu C, Kreuzer E and Hedrick J K. 2017. Learning a deep neural net policy for end-to-end control of autonomous vehicles//Proceedings of 2017 American Control Conference. Seattle, USA: IEEE: 4914-4919[ DOI: 10.23919/ACC.2017.7963716 http://dx.doi.org/10.23919/ACC.2017.7963716 ]

Rosique F, Navarro P J, Fernández C and Padilla A. 2019. A systematic review of perception system and simulators for autonomous vehicles research. Sensors, 19(3):#648[DOI:10.3390/s19030648]

Sampaio R C B, Becker M, Siqueira A A G, Freschi L W and Montanher M P. 2013. FVMS: a novel SiL approach on the evaluation of controllers for autonomous MAV//Proceedings of 2013 IEEE Aerospace Conference. Big Sky, USA: IEEE: 1-8[ DOI: 10.1109/AERO.2013.6497415 http://dx.doi.org/10.1109/AERO.2013.6497415 ]

SatılmışY, Tufan F,Şara M, KarslıM, Eken S and Sayar A. 2018. CNN based traffic sign recognition for mini auton omous vehicles//Proceedings of the 39th International Conference on Information Systems Architecture and Technology-ISAT 2018. Cham: Springer: 85-94[ DOI: 10.1007/978-3-319-99996-8_8 http://dx.doi.org/10.1007/978-3-319-99996-8_8 ]

Schöner H P. 2018. Simulation in development and testing of autonomous vehicles//The 18th Internationales Stuttgarter Symposium. Wiesbaden, Germany: Springer: 1083-1095[ DOI: 10.1007/978-3-658-21194-3_82 http://dx.doi.org/10.1007/978-3-658-21194-3_82 ]

Schwarting W, Alonso-Mora J and Rus D. 2018. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 1:187-210[DOI:10.1146/annurev-control-060117-105157]

Shah S, Dey D, Lovett C and Kapoor A. 2018. AirSim: high-fidelity visual and physical simulation for autonomous vehicles//Hutter M and Siegwart R, eds. Field and service robotics. Cham: Springer: 621-635[ DOI: 10.1007/978-3-319-67361-5_40 http://dx.doi.org/10.1007/978-3-319-67361-5_40 ]

Shu H, Yuan K, Xiu H L, Xia Q and He S. 2019. Construction of basic test scenarios of automated vehicles. China Journal of Highway and Transport, 32(11):245-254

舒红, 袁康, 修海林, 夏芹, 何杉. 2019.自动驾驶汽车基础测试场景构建研究.中国公路学报, 32(11):245-254)[DOI:10.19721/j.cnki.1001-7372.2019.11.025]

Sievers G, Seiger C, Peperhowe M, Krumm H and Graf S. 2018. Driving simulation technologies for sensor simulation in SIL and HIL environments//Proceedings of Driving Simulation Conference Europe 2018. Antibes, France: [s.n.]: 127-130

Skruch P, Długosz M and Markiewicz P. 2017. A formal approach for the verification of control systems in autonomous driving applications//Proceedings of the 19th Polish Control Conference. Kraków, Poland: Springer: 178-189[ DOI: 10.1007/978-3-319-60699-6_18 http://dx.doi.org/10.1007/978-3-319-60699-6_18 ]

Sun P, Kretzschmar H, Dotiwalla X, Chouard A, Patnaik V, Tsui P, Guo J, Zhou Y, Chai Y M, Caine B and Vasudevan V. 2020. Scalability in perception for autonomous driving: waymo open dataset//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2443-2451[ DOI: 10.1109/CVPR42600.2020.00252 http://dx.doi.org/10.1109/CVPR42600.2020.00252 ]

Tettamanti T, Szalai M, Vass S and Tihanyi V. 2018. Vehicle-in-the-loop test environment for autonomous driving with microscopic traffic simulation//Proceedings of 2018 IEEE International Conference on Vehicular Electronics and Safety. Madrid, Spain: IEEE: 1-6[ DOI: 10.1109/ICVES.2018.8519486 http://dx.doi.org/10.1109/ICVES.2018.8519486 ]

Tian Y C, Pei K X, Jana S and Ray B. 2018. Deeptest: automated testing of deep-neural-network-driven autonomous cars//Proceedings of the 40th IEEE/ACM International Conference on Software Engineering. Gothenburg, Sweden: IEEE: 303-314[ DOI: 10.1145/3180155.3180220 http://dx.doi.org/10.1145/3180155.3180220 ]

Tuncali C E, Fainekos G, Ito H and Kapinski J. 2018. Simulation-based adversarial test generation for autonomous vehicles with machine learning components//2018 IEEE Intelligent Vehicles Symposium. Changshu, China: IEEE: 1555-1562[ DOI: 10.1109/IVS.2018.8500421 http://dx.doi.org/10.1109/IVS.2018.8500421 ]

Tuncali C E, Fainekos G, Prokhorov D, Ito H and Kapinski J. 2020. Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Transactions on Intelligent Vehicles, 5(2):265-280[DOI:10.1109/TIV.2019.2955903]

Uricár M, Hurych D, Křížek P and Yogamani S. 2019. Challenges in designing datasets and validation for autonomous driving//Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. Funchal, Portugal: IEEE: 653-659[ DOI: 10.5220/0007690706530659 http://dx.doi.org/10.5220/0007690706530659 ]

Wakabayashi D. 2020. Self-driving Uber car kills pedestrian in Arizona, where robots roam[EB/OL ] .[2020-03-30 ] . https://sites.psu.edu/ist110pursel/2018/03/20/self-driving-uber-car-kills-pedestrian-in-arizona-where-robots-roam/ https://sites.psu.edu/ist110pursel/2018/03/20/self-driving-uber-car-kills-pedestrian-in-arizona-where-robots-roam/

Waltz E and Llinas J.1990. Multisensor data fusion. Boston:Artech house.

Wang G J, Deng W W, Zhang S L, Wang J S and Yang S. 2018. A Comprehensive Testing and Evaluation Approach for Autonomous Vehicles. No.2018-01-0124. SAE

Wei J H, Wang W and Gevorkian A. 2016. Autonomous vehicle simulation system. USA, No.20160314224

Wolter S and Golm F. 2016. Method and system for controlling a vehicle during an autonomous control mode. USA, No.9483927

Wotawa F. 2017. Testing autonomous and highly configurable systems: challenges and feasible solutions//Watzenig D and Horn M, eds. Automated Driving. Cham: Springer : 519-532[ DOI: 10.1007/978-3-319-31895-0_22 http://dx.doi.org/10.1007/978-3-319-31895-0_22 ]

Wotawa F, Peischl B, Klück F and Nica M. 2018. Quality assurance methodologies for automated driving. e and i Elektrotechnik und Informationstechnik, 135(4/5):322-327[DOI:10.1007/s00502-018-0630-7]

Yan C G, Xie H T, Yang D B, Yin J, Zhang Y D and Dai Q H. 2018. Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Transactions on Intelligent Transportation Systems, 19(1):284-295[DOI:10.1109/TITS.2017.2749965]

Yao S W, Zhang J H, Hu Z R, Wang Y and Zhou X L. 2018. Autonomous-driving vehicle test technology based on virtual reality. The Journal of Engineering. 2018(16):1768-1771[DOI:10.1049/joe.2018.8303]

Yu Z P, Xing X Y and Chen J Y. 2019. Review on automated vehicle testing technology and its application. Journal of Tongji University (Natural Science), 47(4):540-547

余卓平, 邢星宇, 陈君毅. 2019.自动驾驶汽车测试技术与应用进展.同济大学学报(自然科学版), 47(4):540-547)[DOI:10.11908/j.issn.0253-374x.2019.04.013]

Yue X, Wu B, Seshia S A, Keutzer K, Sangiovanni-Vincentelli A L. 2018. A lidar point cloud generator: from a virtual world to autonomous driving//Proceedings of 2018 ACM on International Conference on Multimedia Retrieval. New York, USA: IEEE: 458-464[ DOI: 10.1145/3206025.3206080 http://dx.doi.org/10.1145/3206025.3206080 ]

Zhang C, Liu Y H, Zhao D C and Su Y Q. 2014. RoadView: a traffic scene simulator for autonomous vehicle simulation testing//Proceedings of the 17th IEEE International Conference on Intelligent Transportation Systems. Qingdao, China: IEEE: 1160-1165[ DOI: 10.1109/ITSC.2014.6957844 http://dx.doi.org/10.1109/ITSC.2014.6957844 ]

Zhang M S, Zhang Y Q, Zhang L M, Liu C and Khurshid S. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems//Proceedings of the 33rd IEEE/ACM International Conference on Automated Software Engineering. Montpellier, France: IEEE: 132-142[ DOI: 10.1145/3238147.3238187 http://dx.doi.org/10.1145/3238147.3238187 ]

Zhao D C, Liu Y H, Zhang C and Li Y C. 2015. Autonomous driving simulation for unmanned vehicles//Proceedings of 2015 IEEE Winter Conference on Applications of Computer Vision. Waikoloa, USA: IEEE: 185-190[ DOI: 10.1109/WACV.2015.32 http://dx.doi.org/10.1109/WACV.2015.32 ]

Zhou H S, Li W, Kong Z K, Guo J F, Zhang Y Q, Yu B, Zhang L M and Liu C. 2020. DeepBillboard: systematic physical-world testing of autonomous driving systems//Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering. Seoul, South Korea: ICSE: 347-358[ DOI: 10.1145/3377811.3380422 http://dx.doi.org/10.1145/3377811.3380422 ]

Zhu B, Zhang P X, Zhao J, Chen H, Xu Z G, Zhao X M and Deng W W. 2019. Review of scenario-based virtual validation methods for automated vehicles. China Journal of Highway and Transport, 32(6):1-19

朱冰, 张培兴, 赵健, 陈虹, 徐志刚, 赵祥模, 邓伟文. 2019.基于场景的自动驾驶汽车虚拟测试研究进展.中国公路学报, 32(6):1-19)[DOI:10.19721/j.cnki.1001-7372.2019.06.001]