CAREER: Accounting for Measurement Bias and Error in Integrative Data Analysis

This CAREER project will develop a robust and efficient framework for integrative data analysis (IDA). Data sharing activities mark one of the most significant trends in science for the past two decades, establishing many rich public data sets in the social and behavioral sciences. Leveraging diverse data sets allow researchers to obtain stronger statistical power and ask broader research questions.

Using IDA, researchers can pool multiple data sets to obtain more accurate statistical results. However, in the social and behavioral sciences, different instruments usually are used to measure the same concept, which presents a significant challenge for IDA. This project will develop statistical methods and open-source software to address this challenge.

The tools to be developed will allow researchers to harmonize and adjust incompatibility and biases across data sources. In terms of educational activities, the investigator will (a) develop training workshops for applied researchers to increase proficiency in IDA, (b) create an open-access course on psychological measurement to increase public knowledge, and (c) develop a training curriculum to increase the data literacy of high-school educators for underrepresented students.

This research project will enhance IDA infrastructure by proposing a robust and efficient latent variable framework that reduces labor and computational time for combining multiple data sets. Despite immense potential, existing IDA approaches do not provide a unifying framework to address major measurement issues. These issues include the use of incompatible measures across studies, the unreliability of measured scores, and the violation of measurement invariance across different subgroups and time points.

The new data harmonization algorithm to be developed will be the first robust optimization procedure that simultaneously accounts for measurement incompatibility and measurement bias. The project also will develop and validate a statistical procedure, two-stage random-effect path analysis, for analyzing harmonized scores that correctly accounts for measurement error.

Open-source and accessible software will be developed for these new statistical tools, and methodological guidelines will be created that inform when these methods perform well and when cautions should be applied. The methodological tools and guidelines to be developed will allow researchers in the social and behavioral sciences to answer new research questions in many areas.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.