Data Visualisation: Learning From Big Data

The aim of this research project entitled "Data Visualisation: Learning From Big Data" is to answer the question: "how can engineers contend with the current challenges of Big Data, particularly those concerning data volume, velocity and high-dimensionality, so they may more effectively extract meaningful insights from their simulations and experiments?". At present, one of the many difficulties faced by research engineers is that of storing, sending, recovering and viewing extreme volumes of simulation data in productive or cost-effective time frames.

This issue, often referred to in the literature as the "I/O bottleneck", is only going to be exacerbated by continued advances in compute capability rooted in the maturation of hardware accelerated high-performance computing. For example, in fields of study such as Turbomachinery, cutting edge Direct Numerical Simulations (DNS) now yield single-snapshot outputs each demanding tens of GigaBytes of storage space.

With average compute speeds now exceeding current best data transfer rates by nearly an order of magnitude, it is becoming almost impossible to inspect and manipulate such large volumes of data with current techniques and infrastructure. As such, new approaches must be considered.

The initial aims of this project are to investigate, develop, and successfully implement software solutions to the I/O bottleneck problem, focussing evenly on massive ensemble dataset and large singular data volume applications. The preeminent approach is to implement compression as a means of representing large multidimensional volumes of simulation or experimental data in more compact forms.

Neural implicit representations are to be explored as an avenue for achieving extreme data compression without significant loss of information. A practical target is to visualise and ultimately interact with DNS simulation data in real-time within a web-based viewer: a task that is not easily accomplished with current capabilities. Work will also explore training schemes for neural representations, as well as aspects of the decompression process.

Additional research aims are to investigate and apply statistical tools, data science methods, and machine learning (ML) methods to assist in automatic insight generation. The goal of this work will be to lower the demand for both domain-specific knowledge and proficiency in statistics when performing exploratory data analysis (EDA). Finally, stretch goals are to also explore new and disruptive technologies for application in interactive data visualisation.