AI for team collaboration

Purpose and goal:

The aim of the project was to intertwine research in multimodal interaction analysis (EMCA, see Broth & Keevallik, 2020) with interaction design. The project focused on the development of an AI teammate that supports human teams by providing continuous feedback on team dynamics (verbal and non-verbal). Through the project we forged strong links between EMCA and interaction design and demonstrated an example of how video analysis can be drawn upon as an important resource throughout the design process.

Expected results and effects:

We developed a new technique for designing minimal interactive sounds, which can signal that an AI agent is participating in a discussion. We describe the technique in a paper submitted to the THRI journal. Ongoing research was presented at two workshops at major robotics conferences (HRI, ICRA).

We have developed a course that brings together video analytics and interaction design for academics and practitioners, which we taught at the CHI conference. The project team continues writing several academic papers on AI as a teammate and sound design, which are tied to this project. Approach and implementation:

We analyzed and discussed video data of interactions between an AI teammate prototype and volunteer participants in weekly open data sessions. Researchers from various labs at Cornell University participated in the data sessions led by Hannah Pelikan. We held iterative protoyping sessions to design the interaction with a focus on minimal noise, which has not been systematically explored before.

Together we developed a voice-over technique for sound design that combines EMCA with interaction design.