A quality validation toolbox for automotive perception data towards trustworthy AI

Purpose and goal:

In this project, the purpose is to improve the interpretability and reliability of AI systems by better understanding data quality and errors that may occur in data at different stages throughout the AI pipeline. The objective is to develop a quality control toolbox for data collected and consumed by automotive perception systems. The toolbox is end-to-end in the sense that it handles data from its raw format to high level formats such as annotations and AI system predictions.

Expected results and effects:

As a result of this project, we have implemented a first version of the quality control toolbox, Qually. Qually takes data from various sensors at different transformation stages and produces a set of quality metrics for individual data points and collections of data. A planned next step is to improve Qually in terms of its rigorousness, capacity, scalability and completeness.

Moreover, we also plan to improve the set of data properties and quality specifications to further investigate the quality of data and how different types of errors propagate and affect the AI system as a whole. Approach and implementation:

In order to achieve the objective, we first categorized data into four formats - raw format, media format, meta format and annotation format - depending on their interface and the information they carry. For each data format, we defined a set of data quality specifications. We then developed a quality check toolbox given these specifications for data validation and anomaly detection. As an application, we used this toolbox to improve the automated annotations produced by our AI system.