The Next Generation of Learning Machines: A Nonconvex Design

The motivation behind this proposal is the observation that the current gold standard of Machine Learning, i.e., empirical risk minimization with convex regularization (R-ERM), is increasingly unable to meet the demands of a rapidly-growing field. The limitations of R-ERM, which dates back to the 70’s, have tangible ramifications for ML and beyond.

Our recent work has, for the first time, quantified the limitations of R-ERM in a unified framework, and has led us to believe that these limitations can be overcome by CERMON, a new platform for learning with nonconvex regularization.

A detailed understanding of the statistical and computational aspects of CERMON, however, requires the two to be studied jointly, and such interdisciplinary studies are in their infancy.

Indeed, the existing attempts to replace R-ERM are too specialized, limited to “morally” or “nearly” convex problems, or indirect nonconvexity.

The few direct attempts are mired in computational difficulties.We will overcome these challenges and develop a groundbreaking statistical learning theory for CERMON, inspired by the rise of interpolating machines.

In tandem with our pioneering implementation of CERMON, this statistical framework is expected to set a new gold standard for ML.

Our goal of developing a unified platform for nonconvex learning is ambitious, but promises huge impacts within and beyond the data sciences. We will showcase this theory in deep learning, matrix factorization, and inverse probblems.