PlayGround - Embodied Affordance Grounding using Semantic Simulations and Neural-Symbolic Reasoning

While state of the art research can produce computational models of language and vision that enable machines to achieve several tasks, e.g. answer questions about images, localize referential expressions to 2D object boxes, and generate an image from a Natural language description,  one essential task for embodied agents is to be able to discerne impossible goals from possible ones.

For instance, statements “The coffee inside the mug” are clearly discernable from “the mug inside the coffee”.

Thus the ability to reason about affordances, i.e.,  affordability inference as what is discernable is determined by the understanding of affordances themselves.

The overall goal of this project is to contribute novel techniques for affordance inference and for symbol grounding that are based on 1) an integrated symbolic - subsymbolic framework, and 2) a semantic simulation framework.

To achieve its aim PLAYGROUND rests on the hypothesis/assumption that high-level concepts and reasoning processes can be leveraged to improve affordance grounding.

As reasoning about affordance often requires multiple inference steps, it is our hypothesis is that these processes can be simulated. Thus, PLAYGROUND uses simulation as an important element in the grounding process.

With the advent of state-of-the art machine learning methods in generative adversarial networks, we will also create a new kind of experimental platform for affordance inference that involves synthetic data sets.