Integrated modeling of speech, face and gestures for virtual characters

Reproducing the dynamic complexity of verbal and nonverbal expression with computational models has been the objective of the research field for a long time. However, speech and motion synthesis have invariably been treated as separate problems.

In this project, we propose Integrated Behavior Generation (IBG) for virtual characters, a novel approach to jointly model audiovisual modalities (speech, gesture and face) from text in a synchronized manner using a unified machine learning architecture based on probabilistic generative modeling. We also propose high-level style control to enable text-to-behavior synthesis with meaningful contextual parameters.

Our recent achievements winning multiple awards in the areas of spontaneous speech synthesis, speech-driven gesture generation and generative modeling of continuous signals, drive this ambitious approach.

To be able to train fully integrated models, we introduce a comprehensive setup to record and visualize high-precision capture of face and body motion together with speech.

This allows collecting naturalistic and anonymizable recordings of communicative behavior under different realistic contexts eliciting a variety of styles.This project will significantly advance the state of the art in automatic generation of verbal and nonverbal behavior, and we expect IBG to develop into an entire research field of its own, posing new challenges and opportunities for the research community for many years to come.