< Back to previous page

Project

Structure from Semantics

The SfS project (Structure from Semantics) aims at developing a novel approach to 3D reconstruction, starting from a single RGB image as input. This new approach is completely semantics-driven: by analysing the image content, recognizing and localizing the objects therein, an estimate of the 3D of the whole scene will be obtained, building on a large dataset of 3D object category models that can be fitted to the objects depicted in the image. Whereas semantic 3D methods in the literature start from a 3D reconstruction and then add semantics, we start from the semantics, and derive the 3D from there. If successful, such semantics-driven 3D offers several advantages over traditional 3D reconstruction: the output is not just a 3D point cloud or 3D mesh, but a detailed model consisting of 3D objects, each with characteristic shape and material properties, placed in a 3D scene. This goes a step further than current state-of-the-art image understanding based on 2D bounding boxes. Moreover, since full 3D object models are fitted, new view synthesis without gaps in the model caused by occlusions then becomes possible.

Date:1 Jan 2017 →  31 Dec 2020
Keywords:Semantics
Disciplines:Modelling, Multimedia processing