Exploring high-dimensional loss landscapes: Asymptotics for learning algorithms and interacting particle systems arising in data science

Within the mathematical foundation of data science, understanding generalisation properties of deep learning is one of the most important outstanding challenges.

A related challenge, at the intersection of probability, optimisation and data science, is to give precise statements on how learning algorithms, such as stochastic gradient descent and its offsprings, are able to efficiently explore the complex high-dimensional loss landscapes arising in the training of neural networks.

Despite a flurry of activities in recent years, in many ways our understanding of even basic properties of deep learning and learning algorithms are still at its infancy, often based on semi-rigorous studies and folklore.

These are the challenges I will address in this project.The long-term aim of the proposed research is to study generalisation properties in deep learning using tools from probability and analysis, most notably large deviation theory and weak KAM theory.

The main goal is to build a systematic, rigorous framework for studying generalisation, through the lens of interacting particle systems and Lagrangian dynamics.

In particular, to characterise generalistaion properties in terms of large deviation rate functions, and to study the combination of loss landscapes arising in deep learning and properties of trajectories of common learning algorithms.

With the project grant, I plan to hire a PhD student to work with me on the different research questions outlined in the proposal.