SACCADE - Semantic-Aware Compactor for Curating Data Lakes

AI/ML-enabled systems need large amounts of high quality data for training and evaluation. At industrial scale, such datasets are already referred to as data lakes to denote their size. Growing them by adding more data drops is rather simple.

However, determining the added value of the next drop to be inserted to a data lake has become increasingly complicated.In SACCADE, we aim at investigating, developing, and systematically evaluating an automated approach to estimate the added value of new data to be inserted to a large-scale data lake in a computationally efficient way.

We will exploit space-filling curves (SFC) to map multi-dimensional data as well as unstructured large multimodal data to their corresponding, single dimensional representations.

For the latter, we will embody a deep learning (DL) approach to extract semantic-aware key features as input to SFCs.We will use patterns emerging on SFCs to discover already present data samples and also to identify characteristics of still missing data samples in a data lake.

Thereby, it will not only be possible to determine the level of data diversity and quality in existing data lakes, but also scenario identification to improve a data lake will be accelerated.The project is planned to run over 4-years and will use state-of-the-art, large datasets like EuroFOT for performance evaluation.The project provides an automated approach to tackle the important question: "How good are our datasets and when do we have enough data?