Project
Structure from Semantics
The SfS project (Structure from Semantics) aims at developing a novel approach to 3D reconstruction, starting from a single RGB image as input. This new approach is completely semantics-driven: by analysing the image content, recognizing and localizing the objects therein, an estimate of the 3D of the whole scene will be obtained, building on a large dataset of 3D object category models that can be fitted to the objects depicted in the image. Whereas semantic 3D methods in the literature start from a 3D reconstruction and then add semantics, we start from the semantics, and derive the 3D from there. If successful, such semantics-driven 3D offers several advantages over traditional 3D reconstruction: the output is not just a 3D point cloud or 3D mesh, but a detailed model consisting of 3D objects, each with characteristic shape and material properties, placed in a 3D scene. This goes a step further than current state-of-the-art image understanding based on 2D bounding boxes. Moreover, since full 3D object models are fitted, new view synthesis without gaps in the model caused by occlusions then becomes possible.