Few-View 3D Reconstruction with Physics-Based Priors

Few-view 3D reconstruction is a long-standing problem in computer vision and graphics which deals with the problem of using a small number of 2D images to create a 3D representation of a scene. With few views, objects in a scene will generally have unseen parts, making a 'generative' component necessary for a full 3D reconstruction. This project aims to use physics-based priors to help with the missing information necessary for the full 3D reconstruction of a scene.

By incorporating physics-based domain knowledge into the problem of 3D reconstruction, the project aims to obtain more accurate and trustworthy 3D reconstructions, enabling applications in domains such as robotics, AR/VR, and science. Objectives:

The task of using physics-based priors for 3D reconstruction can be separated into a smaller set of research objectives. 1: 3D Representation

Firstly, it is necessary to investigate what data structure for storing the 3D representation of a scene is most appropriate with physics-based priors. The chosen 3D representation is crucial as it affects the quality of shapes and details in reconstructions, while also affecting how many views are required for high fidelity. 3D representations can broadly be considered to be explicit, such as a 3D voxel grid, or implicit, such as storing the 3D reconstruction in a neural network.

An implicit representation can be more flexible, however an explicit representation can be better geometrically restrained, which can be useful for incorporating physics-based priors. 2: Physics-based Priors

The second objective is to explore what physics-based priors are most useful for 3D reconstructions, and may vary depending on what scenes are of interest. For example, if considering the reconstruction of natural phenomena such as clouds, then priors from atmospheric physics may be useful, such as assuming that clouds are generally shaped to be horizontally long and vertically short.

For indoor scenes, which mainly consist of man-made objects, other priors may be found to be more useful, such as assuming that objects contain symmetries, or that objects are built of simpler geometries such as flat surfaces. 3: Training Process

The third key objective is to explore how a model with physics-based priors can be best trained, and would focus on two main aspects, the data, and the training supervision. When considering what data to train a model on, it is important to consider what ground truths the data source provides. For example, when incorporating physics-based priors based on geometry, it may be useful to use data which also provides depth information.

For training supervision, it is worth considering if the physics-based priors should be directly incorporated into the model, or if the priors should be implicitly enforced through the training supervision. Novelty:

Existing methods in few-view reconstruction commonly train a model such that it implicitly learns how to reconstruct unseen parts of objects. However, this requires large amounts of 3D data (which is not readily available), for good accuracy on a wide range of scenes. While 2D image data is readily captured by a camera, 3D data requires specialised scanning equipment or complicated data collection processes.

Previous works have partly tackled this by creating synthetic 3D data, However, this is still fundamentally limited by how many high quality 3D assets can manually be designed by 3D artists (in a time consuming process).

This project instead aims to reduce the amount of data necessary by using physics-based priors to bridge the gap in missing information. Furthermore, incorporating physics-based priors can be implemented in parallel with data-based research.

For EPSRC: This project falls within the "EPSRC Image and vision computing" and the "EPSRC Graphics and visualisation" research areas. It is also relevant to the "EPSRC Artificial intelligence and robotics" research