Resource-efficient optimization for ML: the next frontier

Machine learning (ML) and artificial intelligence, popularized by applications like self-driving cars and GPT-4, are capturing the imagination of people worldwide.

Somewhat surprisingly, the rapid recent advances in large language models cannot be attributed to fundamentally new ideas, but are rather due to the use of larger neural networks and more computational resources for training. Although scaling up ML models is enabling unprecedented advances, training large models has become extremely expensive.

For example, GPT-3 training was estimated to cost $1.65 million.

The number of parameters in large language models has been doubling every 4 to 8 months, and present models have several hundreds of billions of parameters. Extrapolating current trends, the training cost of the largest AI model in 2026 would be more than the total U.S. GDP. Growing model and data sizes to such scales make current training strategies unsustainable!

To address these challenges, this proposal aims to develop optimization theory and algorithms that improve the resource efficiency of machine learning pipelines.

Specifically, we aim to (a) develop novel algorithms that allow for faster and more resource-efficient training on parallel compute resources and (b) introduce efficient schemes for model initialization and dynamic training sample selection that reduce the total training times. Together, such results will allow to train larger machine learning models, faster.