Robust Learning via Mutual Information Regularization

Recent results in information theory show that the generalization error of stochastic learningalgorithms is bounded by the amount of information that an algorithm (e.g., a neural network) stores in its parameters about the training data.

The bounds suggest that in order to obtain learning algorithms that generalize well to unseen data, the information stored in the parameters needs to be controlled to find the best tradeoff between empirical risk minimization and minimization of the generalization error. How this can be achieved is an open problem that we address in this project.

The goal of this project is to develop new theoretical tools that are required for crafting learning algorithms that perform well, generalize well, and are robust, and that provide new insights into how machines learn.

To reach this goal, the objectives of this project are (1) to develop a new family of neural networks that are derived from parity-check matrices of sparse-graph codes and allow us to relate the amount of information stored to their design parameters, (2) to develop tools for tracking the information in the parameters during training, and (3) to extend these tools to multitask learning.

To achieve this, we combine tools from information and coding theory with Bayesian neural networks.

The work in this project will be carried out by one PhD student over the duration of four years, supervised by the applicant.