CAREER: Perturbation Methods for Quantifying Uncertainties in Machine Learning Models

In applied machine learning and statistics, it is common practice to search through many different models before estimating a best model: one that is simple to explain, while still providing good predictive performance. Over the past decade, several methods have emerged which first estimate a model from a range of choices, and then fit the estimated model to extract useful trends and predict future outcomes.

However, predictive accuracy, on its own, has limited explanatory value and point estimators with high uncertainties may lead to poor replicability down the line. Yet, most such models, supervised or unsupervised, lack uncertainties for the related estimators. This project introduces a new class of perturbation methods to quantify uncertainties in machine learning models, with various applications in regression, classification, and dimension reduction.

The research plans have three main goals. The first goal is to develop methods that can be used with different types of data and are not limited to specific models. The second goal is to ensure that the methods can be scaled and applied to decentralized datasets on multiple machines.

The last goal is to create versatile methods that can be used with different estimation techniques. An overarching goal is to allow researchers to apply these techniques to various data types and forms, without being constrained by unrealistic assumptions or limited methods for model estimation. The project's educational and outreach plans are closely tied to its research plans.

The project will help the PI conduct summer training programs with K-12 outreach, develop new curricula, and broaden participation of underrepresented groups in the field.

This project aims to develop methods for attaching uncertainties to the outputs of model estimation methods. The research agenda is structured into three main aims. In the first aim, the project will introduce distribution-free methods that can quantify uncertainties in a flexible class of semiparametric models.

In the second aim, the project will develop distributed methods that use decentralized data from a cluster of nodes to quantify uncertainties in an estimated model. These methods will need only basic, aggregated statistics from each node and will be accompanied by communication-efficient algorithms. In the third aim, the project will focus on developing perturbation methods as a versatile approach for uncertainty quantification that can be used in a wide range of model estimation algorithms, both supervised and unsupervised.

To achieve these aims, the project will use perturbation to exploit a link between the geometric properties of estimators and their underlying probability, and will employ an integrated approach using mathematical statistics, probability theory and optimization. Throughout the project and after its completion, the methodology and open-source software will be applied to improve biomedical decision-making and replication in health studies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.