Information, Generalization and Optimization

This project will contribute to the mathematical analysis of modern (deep) learning.

There has been a recent surge in pursuing information-theoretic bounds for the learning generalization error, and we have also contributed to this development.

However, the ultimate goal is arguably to minimize the statistical excess risk, and therefore we also need to better understand the so-called optimization error.

Here there is a tradeoff between generalization and optimization, parameterized by the information flow from the training data to the learned hypothesis.

As far as we know, we are the first to propose to analyze this tradeoff, with an overall novel focus on bounding the optimization error using information-theoretic tools. Thus, the proposed work can be considered as high risk but also with the prospect of high reward.

Via two toy-examples we demonstrate that the tradeoff exists and that it should be possible to bound and characterize it. The tradeoff between generalization and optimization applies primarily to supervised learning. However, similar considerations are also valid in reinforcement learning.

In preliminary work, we have contributed some of the first information-theoretic bounds to the minimum Bayesian regret, and we plan to continue this development also with a new focus on risk-aware learning.

We will furthermore strive to analyze privacy preserving learning algorithms, since the information metrics we consider are commonly used also to quantify privacy leakage.