Beyond 3D Points in Sparse Visual Mapping

Building internal representations of the world is a requirement for any system that needs 3D spatial awareness.

For example, it enables autonomous agents (e.g. cars or drones) to localize and map their environments such that they can safely navigate and operate.

Similarly, it allows augmented reality applications to keep a persistentent and shared coordinate frame to anchor the virtual content.

For cross-session persistency, there is a need for off-line mapping solutions which can aggregate large quantities of imagery and sensor data into compact map representations.Current sparse 3D reconstruction frameworks represent the scene as a collection of triangulated 3D points, essentially sampling the surface at a discrete set of locations.

We argue that many failures stem from this representation which only encodes very low-level and local properties.

This lack of global context in the reconstructions result in models that are locally accurate but drift on a larger scale.In this project we will develop the next generation of sparse reconstruction frameworks that avoid these issues. The key idea is to jointly build a connected wireframe that captures the coarse layout of the scene.

The wireframe model will allow us to encode non-local geometric priors such as orthogonality, parallelism and co-planarity.

Our envisioned reconstruction pipeline relies on the sub-pixel accurate keypoints to get precise poses locally, and the rigidity of the wireframe to correct for global drift.