Modular multilingual models

Natural language processing (NLP) has seen tremendous progress recently through the use of complex neural network models that learn effectively from very large amounts of data, but the use of data-intensive large-scale models also gives rise to challenges. How can we provide access to NLP technology without access to super-computers?

How can we bring NLP to languages with more limited resources than English?

A promising approach to the second problem is to train multilingual models, by pooling data resources from multiple languages with the hope that they will mutually reinforce each other.

Unfortunately, the effect observed has often been the opposite, with quality decreasing as more and more languages are added, an effect known as the curse of multilinguality.

This project explores modularization as a key to making models for multilingual NLP more effective, by delivering higher quality for more languages, and more efficient, by requiring less resources to do so.

We will focus on techniques related to model architecture (static modularization) and to training and tuning procedures (dynamic modularization).

In addition, we will study the interplay between these techniques and methods for improving efficiency, in particular model compression.

The specific aims of the project are to develop multilingual models that (a) perform on par with comparable monolingual models, (b) can be dynamically extended with new languages, and (c) can be compressed for efficient deployment.