Validation of a Virtual Still Face Procedure and Deep Learning Algorithms to Assess Infant Emotion Regulation and Infant-Caregiver Interactions in the Wild

“This study is part of the NIH’s Helping to End Addiction Long-term (HEAL) initiative to speed scientific solutions to the national opioid public health crisis. The NIH HEAL Initiative bolsters research across NIH to improve treatment for opioid misuse and addiction.”

Moment-to-moment infant-parent interactions are a central context in which infants learn to regulate emotions. Investigating infant-parent interactions in which emotion regulation unfolds is particularly important for infants at risk for emotion dysregulation and/or relationship disturbance, including infants with prenatal substance exposure. Yet, current state-of-the art methods to assess infant emotion regulation and infant-parent interaction predominantly rely on brief laboratory tasks. These procedures pose burdens on participants, especially families

experiencing demographic and psychosocial risk, and place limits on generalizability and ecological validity of findings. Technological advances in (a) machine learning methods, including deep learning approaches that mine for complex patterns in raw unlabeled data, and (b) wearable sensors have the potential to transform our ability to capture infants’ moment-to-moment emotional experiences in their real-world environments, while also lowering burden on families participating in infant research.

With these issues in mind, we will develop next-generation methods to assess infant emotion regulation and infant-parent interaction. In doing so, we will use

LittleBeats, an infant multimodal wearable device developed by our team, to collect time-synced data on infant and parent vocalizations (via microphone), infant motor activity (via motion sensor), and infant cardiac vagal tone (via electrocardiogram [ECG]) for extended periods of time (~8-10 hours per day) in the home. We propose three specific aims.

First, we will validate a virtual visit protocol for the gold-standard Still Face Paradigm, which is typically conducted in a laboratory setting, for assessing emotion regulation among infants during the first year of life. Second, we will validate multimodal deep learning algorithms to detect infant emotional states in real time

using LittleBeats audio, ECG and motion data. Third, we will validate deep learning algorithms to detect and label vocalization types of infants (babble, fuss, cry, laugh) and parents (infant-direct speech, adult-directed speech, sing, laugh), which create the build blocks of infant-parent vocal interactions, such as turn taking. By bringing together innovative wearable technology with cutting-edge deep learning algorithms, we aim to advance understanding of the mechanisms through which prenatal substance exposures contribute to adverse outcomes.

Further, prenatal substance exposure is a heterogeneous phenomena that transacts with environmental risk and

protective factors, thereby making a one-size-fits-all approach ineffective. By monitoring moment-to-moment changes in infants’ emotion regulation, combined with deep learning algorithms that detect and classify infant-parent interactions during moments when infant show signs of distress, the proposed methods have the potential to transform our understanding of the dynamic processes through which prenatal substance exposure leads to poor outcomes and pinpoint protective factors that promote optimal development.