Research visit to Mila-Quebec AI

Purpose and goal:

The main goal of this project was to initiate a research collaboration between RISE and the Mila-Quebec AI Institute, primarily on the research topic of continual learning. This goal was accomplished through performing an experimental study on the topic of continual language model training. An additional goal was to initiate a communication channel between RISE and Mila-Quebec AI, leading to a closer future collaboration in the form of knowledge transfer workshops and innovation projects. This interaction is planned to continue even after the end of the research visit.

Expected results and effects:

The main outcome of this project is a research study of the catastrophic forgetting effect of continual training a laguage model on different languages. The results of this study indicate that the model forgets previously learned knowledge in varying degree, depending on the similarity between languages. Furthermore, this research visit gave the opportunity to researchers from the two organisations to brainstorm on new research ideas, that can lead to common innovation projects in the future.

Approach and implementation:

The main project activity was a four-month research visit of a RISE employee to Montreal, where Mila-Quebec AI is located. During the research visit, the travelling employee collaborated with researchers from the host organisation on the topic of continual learning for language models. Apart from hands-on research collaboration, the travelling employee had the opportunity to participate in reading groups on various machine learning topics and present in both internal and external seminars organised by Mila-Quebec AI.