Testing Self Driving Cars - 1 : Pegasus Project

Pegasus is a geovernment sponsored project aiming to provide new methods for the verification and validation of highly automated driving functions (SAE Level 3+). In other words, It's an attempt to answer the question "How safe is a self driving car?". In this article, I will summarise important points in the original Pegasus paper first section.

1. Abstract

In past years automotive companies focused on automated driving functions since there is a public interest. But, the development was more focused on functional, not on neither test nor validation and verification.

And for testing of automated driving functions, distance based testing is currently used.

The required tests distance for an automated driving function within the highway pilot odd is approximately 6.2 Billion kilometers to show that the automated driving function is twice as good as a human driver. (Wachenfeld & Winner, 2015)

This distance is obviously not realistic to drive and test the vehicle.

One another option is scenario based testing. This gives the opportuinity to do the testing in a more structured way instead of distance based testing with random test cases.

What is the human driving performance within highway-pilot ODD?

To answer this question, first, the accident database GIDAS is analysed and picked the accidents lie in the ODD. Then simulator studies were performed to derive a indicator model for human driver for selected scenarios.

How to test AD-functions within the highway-pilot ODD?

Execution and evaluation of scenarios were done in simulation, on test track or in real traffic. This work requires these tasks to be completed:

• Systematic scenario generation process
• Scenario definition syntax
• Calculation of criticality parameter (KPI) for scenarios

How is a scenario modelled in scenario based testing?

Six independent layers. (A.k.a 6-layer model)

1. Road geometry
2. Road furniture and rules (traffic signs...)
3. Temporary modifications and events (construction sites, etc...)
4. Moving objects (other cars, pedestrians...)
5. Environmental conditions (weather, sun ...)
6. Digital information (map)

Restricting this large parameter space with the ODD provides a full test space of the system. But it is still a very large parameter space to cover even after filtering with ODD. To overcome this issue, Pegasus method defines a logical scenario that some parameters are fixed and some parameters are variable. One example is fixind the layer 4, with a 'cut-in' scenario. In the cut-in scenario, the constellations of the moving objects and the ego vehicle is fixed. And the rest of the parameters can be varied.

How to test a logical scenario?

The overall goal is to identify collision relevant concrete parameter areas in the logical scenario's whole parameter space.

The parameters that contribute to the testing goal most is identified at sensitivity analysis phase. Based on this identification, the number of relevant parameters are reduced dramatically. Empirical studies showed that the numbers of most important parameters are in the range of 10 and 20. This reduces the parameter space which is needed to be covered. Pegasus database provides real world data to limit parameter ranges and reduce the paramter space such as:

• concrete parameter sets for representative accidents
• observed critial constellations for the logical scenario
• distributions and correlations of parameters in real life

Based on the stochastic variation within the parameter space, concrete parameter sets can be created automatically. Then, each concrete parameter set corresponds to a concrete scenario and vice versa. A black-box simulation run is performed with a fail or pass result for each concrete parameter set. The main goal is to find concrete parameter sets with failed (ego crashed) results. After that, critical cases get tested in real cars.

In summary, once a logical scenario is defined, free parameters are varied by a sampling schema and concrete scenarios are created. By running each concrete scenario in simulator, critical parameter set areas are identified. Some critical test cases are tested on proving grounds. Because simulation has limitations especially in sensor models and extreme driving dynamics. It is not possible to test edge cases in real traffic, they have to be performed on proving grounds. In contrast, in real traffic, the behavior of the driving function is tested. The main goal is to catch "suprises" like new parameters, new scenarios.

---

Summary

• Exhaustive testing needed to test the automated driving functions
• There are two approaches:
  • Distance based testing (2.6 billion kms needed)
  • Scenario based testing
• Define a logical scenario
• Automatically create concrete scenarios based on stochastic variation within the parameters in logical scenario
• Run each concrete scenario in simulation
• Identify risky parameter set areas
• Test on proving grounds critical cases
• Test in real traffic for new parameters and new scenarios