CAREER: Physical Object Modeling for Intelligent Systems

Recent advancements in artificial intelligence (AI) and computer vision have shown the possibility of building machines that can see and explain what is in an image or video. However, for machines to physically interact with the world just like humans, they need to understand more than “what is where” but also that the world contains physical objects with distinct properties, such as their material, shape, density, and multi-sensory characteristics (how they sound and feel).

This problem of physical object modeling remains largely ignored in current AI systems. This project aims to introduce systems and methods that may allow us to capture and model diverse, multi-sensory, physical objects to help intelligent machines better understand and interact with the world. Researchers will integrate findings from this research into course development, student advising, and community infrastructure development; they will also partner with educational and non-profit organizations to teach AI, vision, graphics, and robotics to underrepresented students.

In this project, investigators will focus on the modeling and extensive applications of multi-sensory physical objects. At the core of the technical plan is a physics-based object representation, which encodes physics-based, object-centric priors in the world, including the notion of object texture, material, geometry, articulation, physical properties, and multi-sensory characteristics.

Centered around such representations, the team of researchers will develop (1) new systems that exploit integrated hardware and learning to capture datasets of physical objects; (2) new methods that leverage these objects for computer vision, graphics, and robotics tasks, such as multi-sensory perception and reconstruction, object and scene generation and editing, dynamics modeling and simulation, and robotic interaction and manipulation; (3) new paradigms that study the neurocognitive bases of physical object perception and interaction, with the long-term potential of providing intelligent machines with more human-like capabilities.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.