Dorsal Stream Robot Vision

This project will improve robot vision by exploiting recent findings on the coupling of action and visual perception in humans.Deep machine learning has improved robot vision in tasks such as object detection and recognition. In other tasks such as structure-from-motion (SfM) and vision based robot control, such benefits are yet to come.

Why is it that these applications struggle to reap the benefits of the deep learning revolution?The human visual system handles recognition and visually guided actions using two separate processing streams, called the ventral and dorsal streams.

The ventral stream handles static tasks such as object and face recognition, and is similar in structure to current deep learning.

The dorsal stream, on the other hand, addresses dynamic visual tasks, such as grasping, catching objects, and driving a car. Such tasks rely on sequences of percepts, and need to handle percepts that are distorted by the task at hand. For distortions caused by the task, details of the task can aid perception.

Further, the distortions often contain important cues, and in this project we will use such action-related distortions as a perceptual cue.We will design machine learning architectures that exploit action-coupled cues from video.

For live training we will use self-supervision on bodycams under realistic tasks, together with action proxy signals from inertial measurement units (IMUs). Applications include robot guidance and structure from motion on body-camera videos.