Learning task and motion planning for mobile manipulation robots.

Context

The range of tasks that could be accomplished with a mobile robot capable of manipulating its surroundings is vast. Because of this, traditional methods to train robots are unfeasible for general purpose robotics. In industrial settings, the environment can be adapted for robots; there are a limited number of tasks to be performed and parts can be precisely located such that repeated actions lead to repeated outcomes.

Outside of these controlled areas, a robot must be able to work within an environment designed for and occupied by humans. No guarantees can be made that anything is where it should be and blindly repeating actions that worked previously can lead to potentially dangerous consequences. General robotics requires a far more flexible pipeline than its industrial counterparts.

Aims and objectives

This project aims to construct this pipeline from current state-of-the-art AI models that each solve part of the problem. The goal is to make this pipeline modular so that swapping to the newest models is as easy as possible, which is essential given the rate of advancement within these areas. Some of the key systems within the pipeline are discussed below:

- Vision - Required to identify objects and estimate their pose for future manipulation. The vision system can also be used to spot hazards and for visual confirmation that a task has been successfully completed. There is a trade off to be made between models that are limited in what they can detect and models that are far slower, but can recognise a much more broad array of items.

- Task Planning - A generalised robot needs to be able to take what it is told to do and break it down into a set of instructions that can be followed to complete the task. This is traditionally done by human experts, but this does not work for arbitrary tasks. Large language models demonstrate good knowledge about a wide array of tasks and are capable of breaking them down into simple steps.

- Manipulation - In settings designed for humans, a robot needs to be able to interact with its environment in a similar manner to us. Industrial robots interact with a limited number of items and can use different end effectors to achieve different goals. A general robot needs to interact with many unique objects and be able to use tools in a similar manner to humans. This requires task orientated grasping and a way to generate motion plans for different actions.

- Locomotion - Mobile robots need to be able to navigate around dynamic environments where there could be obstacles that must be avoided. While this will need to be included in the full pipeline, it is not a focus of this project.

There are many more systems required for a functional and reliable robot. However, these are the main areas of focus for this project. Novel research methodology Beyond development of the pipeline, the planned research includes:

- Vision - Explore the use of fast and broad vision models in tandem, for fast detection of common objects while still being able to find uncommon items.

- Task planning - Leverage large language models to break down a task into a set of instructions that can be understood by a robot. This poses a challenge with dealing with the models being wrong or missing crucial information.

- Motion planning - To generate a mapping between actions and the required motions, one path to explore would be to create a set of parametric motion paths that can be chained together, and creating a mapping between actions and paths. A second option would be to create a generative AI model that can produce a motion plan when given an action as a prompt.

Both of these options would need to be intergrated with the vision system in order to locate the correct objects to interact with and avoid any obstacles.

This project falls within the EPSRC Artificial intelligence and Robotics research area. This is an Industrial CASE Studentship in collaboration with Siemens.