Quality Metrics and Oracles for Autonomous Vehicles Testing

The race for deploying AI-enabled autonomous vehicles (AVs) on public roads is based on the promise that such self-driving cars will be as safe as or safer than human drivers. Numerous techniques have been proposed to test AVs, which however lack oracle definitions that account for the quality of dr...

Full description

Saved in:
Bibliographic Details
Published in2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) pp. 194 - 204
Main Authors Jahangirova, Gunel, Stocco, Andrea, Tonella, Paolo
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.04.2021
Subjects
autonomous vehicles testing
Conferences
Correlation
Filling
Measurement
mutation testing
Roads
self-driving cars
Software testing
Systematics
test oracles
Online AccessGet full text

Cover

More Information
Summary:The race for deploying AI-enabled autonomous vehicles (AVs) on public roads is based on the promise that such self-driving cars will be as safe as or safer than human drivers. Numerous techniques have been proposed to test AVs, which however lack oracle definitions that account for the quality of driving, due to the lack of a commonly used set of metrics.Towards filling this gap, we first performed a systematic analysis of the literature concerning the assessment of the quality of driving of human drivers and extracted 126 metrics. Then, we measured the correlation between such metrics and the human perception of driving quality when AVs are driving. Lastly, we performed a study based on mutation analysis to assess whether the 26 metrics that best capture the quality of AV driving according to the human study can be used as functional oracles. Our results, targeting the Udacity platform, indicate that our automated oracles can kill a high proportion of mutants at a zero or very low false alarm rate, and therefore can be used as effective functional oracles for the quality of driving of AVs.
DOI:10.1109/ICST49551.2021.00030