StreetWise: Accelerating Automated Driving with advanced scenario-based safety validation
In cooperation with
Siemens, AVL, Daimler Truck
The future of automated driving depends on reliable safety validation of vehicles to ensure that they operate safely in all traffic situations. TNO StreetWise is an advanced method to describe real driving conditions and their variations and store that in a scenario database to enable scenario-based testing and validation. StreetWise brings safe automated driving one step closer.
With ever more advanced Automated Driving Functions (ADFs), safety validation becomes more important and complex. Given the high number of possible scenarios in which an ADF needs to be tested, virtual simulation – in addition to proving-ground and field testing – is essential to discover if safety requirements are met with reasonable efforts and throughput time. To optimally calculate the safety risks, virtual simulations need to be based on realistic test cases and characteristics. Because TNO StreetWise uses public road driving data as input for scenario identification and storage, and test cases are based on such scenarios, the virtual test cases are representative for real-world scenarios. To ensure a safe implementation of Automated Driving (AD) technologies and to achieve a positive risk balance, TNO offers innovative risk quantification methods and scenario statistics.
Advanced safety methodology
Based on TNO’s extensive experience with vehicle safety assessment, the TNO StreetWise methodology addresses operational safety, which is an extension of functional safety (ISO26262) and safety of the intended functionality (ISO/DIS 21448:2021 SOTIF).   . After all, the automation technology takes over tasks from the human driver and should be prepared for all situations in the operational domain it is designed for (ODD). This advanced methodology is substantiated by a series of scientific articles.
Safer Automated Driving
Challenges for the tested systems are determined and quantified by combining systematic descriptions of the vehicle’s surroundings in scenarios, with systematic (statistical) analysis of the vehicle behaviour in such scenarios. By using statistical data from the exposure to different scenarios, we can quantify risk based on these simulations  . By doing so, our industrial clients can make significant steps towards safer Automated Driving. Authorities responsible for type approval can use TNO StreetWise’s quantified risk estimation to determine the safety of a vehicle before allowing it for deployment onto the public road.
Efficient and flexible pipeline
The TNO StreetWise pipeline automatically processes your driving data into activities, scenarios, parameters and statistics. New types of scenarios can be added quickly using the underlying framework. Python algorithms, running in a cloud environment, ensure flexibility and scalability The pipeline is ready for connected vehicles as well (V2X, I2V). A webGUI provides easy access to the scenario database for extraction of OpenSCENARIO test cases, including the related Euro NCAP tests. Test automation is supported through an API, as successfully demonstrated with AVL ModelConnect, Vires VTD and Siemens Simcenter PreScan. Sensitive driving data can be processed at the client’s premises.
Accelerating safety validation
Many of our partners are working on extensive field-testing, data collection and scenario detection for Automated Driving. By collaborating and securely exchanging scenarios and statistics using Secure Multi-Party Computation (MPC), TNO manages to accelerate safety validation for our partners. At the same time, this technology allows us to fine-tune data collection campaigns and benchmark the databases of our clients.
Our clients benefit from the extensive TNO network of partners. For example, we are champion in a certified Euro NCAP test body, and working on the development of the Cyclist-AEB test protocol (CATS). TNO is also working on the way how Euro NCAP can use scenario databases. The list of partners in B2B and EU projects includes most of the CCAM community (e.g., SAKURA , NTU CETRAN (Singapore), AVL , Siemens, Itility, Magna , IDIADA , BMW , Volvo , Continental , ika  , Virtual Vehicle , and VEDECOM).
PUMAS: a shared scenario database without sharing sensitive data between partners.
How do you test the safety of automated driving using scenarios generated from real traffic data from multiple partners? The PUMAS project (January 2018 - June 2020) has shown us that it is possible to build a shared scenario database without sharing sensitive data between partners. The common collection of traffic scenarios offers unparalleled completeness of a scenario database for the development and release of automated vehicles.
The PUMAS project is a collaboration between Magna Steyr Fahrzeugtechnik and AVL List (both Austrian), Siemens SISS (Netherlands/Belgium), and a Japanese OEM. Each partner gave TNO access to hundreds of hours of motorway driving data. TNO subsequently used Microsoft Azure to process this data in the StreetWise scenario pipeline, yielding more than 74,000 observed traffic scenarios. TNO itself contributed by offering two driving datasets.
For each dataset, TNO’s team developed a converter to a standard, internal data format, which included filtering and configuration. In doing so, the StreetWise algorithms and file conversions in the rest of the pipeline became applicable to each of the datasets. This has generated a highly scalable solution. Although there are many variants in the sensor configurations of different parties’ vehicles, the project showed that conversion to a common format is possible. This has enabled the generation of scenarios based on a heterogeneous fleet.
A central question in scenario-based testing is: ‘Are we sufficiently representative of the diversity on the street?’ This question was answered using a number of completeness indicators, such as scenario counting (sufficient observations of a scenario category for statistics), scenario exposure (number of observations per hour), and the stability analysis of parameters (AMISE).
The algorithms used to generate the scenarios were implemented in a Microsoft Azure environment. An online environment, where partners could log in and view test cases from the StreetWise database, was also created. Via an API, simulation tools such as Vires VTD and Siemens PreScan can retrieve test cases in OpenSCENARIO format and use them for simulations. This also works for test management tools such as Model.Connect and HEEDS. The PUMAS project has not only demonstrated the feasibility of a shared scenario database using data from multiple partners; it has also generated practical insights and experience related to scenario-based virtual security validation. Partners indicated that the project has helped them transform scenario-based testing from a concept to a practical methodology, including tooling.
More about this project
- De Gelder, E., Paardekooper, J.-P., Khabbaz Saberi, A., Elrofai, H., Op den Camp, O., Kraines, S., Ploeg, J., De Schutter, B., "Towards an Ontology for Scenario Definition for the Assessment of Automated Vehicles: An Object-Oriented Framework," in IEEE Transactions on Intelligent Vehicles, vol. 7, no. 2, pp. 300-314, June 2022, DOI: 10.1109/TIV.2022.3144803.
- Op den Camp, O., van de Sluis, J., de Gelder, E., and Yalcinkaya, I., “Generation of Tests for Safety Assessment of V2V Platooning Trucks”, in 27th ITS World Congress, Hamburg, 2021.
- Van de Sluis, J., Op den Camp, O., Broos, J., Yalcinkaya, I., and de Gelder, E., “Describing I2V Communication in Scenarios for Simulation-Based Safety Assessment of Truck Platooning.” Electronics 10, 1 (2021), https://doi.org/10.3390/electronics10192362.
Zlocki, A., Op den Camp, O., Arrúe, Á., Taniguchi, S., Umeda, M., Watanabe, S., and Antona-Makoshi, J., “Towards the Harmonization of Safety Assessment Methods of Automated Driving: SAKURA – SIP-adus - HEADSTART, White Paper” December 9, 2021.
De Gelder, E., Elrofai, H., Khabbaz Saberi, A., Op den Camp, O., Paardekooper, J.-P., and De Schutter, B., “Risk Quantification for Automated Driving Systems in Real-World Driving Scenarios”, IEEE Access 9, vol. 9, pp. 168953-168970, 2021, DOI: 10.1109/ACCESS.2021.3136585.
- “ISO/TR 21934-1:2021 Road Vehicles — Prospective Safety Performance Assessment of Pre-Crash Technology by Virtual Simulation — Part 1: State-of-the-Art and General Method Overview.” https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/07/64/76497.html
- De Gelder, E., Manders, J., Grappiolo, C., Paardekooper, J.-P., Op den Camp, O., and De Schutter, B., "Real-World Scenario Mining for the Assessment of Automated Vehicles," 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020, pp. 1-8, DOI: 10.1109/ITSC45102.2020.9294652.
De Gelder, E., Op den Camp, O., and de Boer, N., “Scenario Categories for the Assessment of Automated Vehicles – version 1.7”, Technical Report, Nanyang Technological University, Singapore: CETRAN, 2020. http://cetran.sg/wp-content/uploads/2020/01/REP200121_Scenario_Categories_v1.7.pdf.
Kalisvaart, S., Slavik, Z., and Op den Camp, O., “Using Scenarios in Safety Validation of Automated Systems.” In: Leitner, A., Watzenig, D., Ibanez-Guzman, J. (eds) Validation and Verification of Automated Systems. Springer, 2020, Cham. https://www.springer.com/gp/myspringer/chapters.
Paardekooper, J.-P., van Montfort, S., Manders, J., Goos, J., de Gelder, E., Op den Camp, O., Bracquemond, A., and Thiolon, G., “Automatic Identification of Critical Scenarios in a Public Dataset of 6000 km of Public-Road Driving”, in 26th International Technical Conference on the Enhanced Safety of Vehicles (ESV), Eindhoven, 2019.
Elrofai, H., Paardekooper, J.-P., de Gelder, E., Kalisvaart, S., and Op den Camp, O., “Scenario-Based Safety Validation of Connected and Automated Driving.”, Vision Paper TNO, 2018.
https://publications.tno.nl/publication/34626550/AyT8Zc/TNO-2018-streetwise.pdf (pdf) .
- Op den Camp, O., van Montfort, S., Uittenbogaard, J., and Welten, J., “Cyclist Target and Test Setup for Evaluation of Cyclist-Autonomous Emergency Braking”, in International Journal of Automotive Technology, Vol. 18, No. 6, pp. 1085-1097, 2017. DOI: 10.1007/s12239−017−0106−5
- EU H2020 Hi-Drive (2021-2025): Large-scale pilot for prolonged automation without interruptions (continuous ODD). Data analysis, edge cases, simulation.
- TKI StreetWise Plus (2021-2022): Extension of scenario database for multi-actor simulation, I2V messages and roadside data sources. With Siemens and Itility.
- EU H2020 ARCADE (2018-2022): Coordination and support project for CCAM projects. Stakeholder workshops on common evaluation methodology (Nov 2020) and edge cases (May 2021).
- EU H2020 Headstart (2018-2021): Scenario-based safety assessment methodology including V2X, localisation and cyber security. Detailed methods for scenario selection and allocation to test methods. Use case truck platooning.
- TNO PUMAS (2018-2020): Multi-partner scenario database project with AVL, Magna, Siemens and a Japanese OEM. Proof of concept of sharing scenarios from diverse driving datasets for 1,000 hours of data. Including completeness metrics.
- EU H2020 L3Pilot (2017-2021): Large-scale pilot for L3 automation. Data analysis and impact assessment for Jaguar Land Rover and Aptiv. Evaluation of lane-change detection.
- EU ECSEL ENABLE-S3 (2016-2019): Virtual verification and validation of automated systems. Methodology, scenario working group, valet parking use case, image-guided surgery use case, dataset inventory.