Scene representation using image processing

A system configured to obtain a mesh representation of a scene,

• wherein the mesh representation comprises a plurality of polygons defined by respective vertices associated with an in-plane position, the in-plane position being in a plane comprising a first dimension and a second dimension, and the vertices having an associated depth value in a third dimension different from the first dimension and the second dimension,
• and the system comprises:
• an in-plane position estimation network configured to process image data representative of an image of the scene to estimate the in-plane positions associated with respective vertices of the mesh representation;
• and
• a depth estimation engine configured to:
• receive the in-plane positions and the image data;
• and
• optimise a cost function comprising a term dependent on the image data and the in-plane positions, to estimate the associated vertex depth values for the respective vertices of the mesh representation.

A method of training an in-plane position estimation network, the method comprising, for a training iteration:

• obtaining a set of training samples from a set of training data, wherein each of the training samples comprises image data representative of a scene and a ground-truth depth map of the scene;
• for each given training sample in the set of training samples:
• processing the image data for the given training sample using a neural network architecture of the in-plane position estimation network to obtain estimated in-plane positions associated with respective vertices of polygons of a mesh representation of the scene, wherein the estimated in-plane positions are in a plane comprising a first dimension and a second dimension;
• using the estimated in-plane positions to obtain a predicted depth map of the scene, wherein the predicted depth map comprises depth values associated with the respective vertices in a third dimension different from the first and second dimensions;
• and
• computing a loss function based on a comparison between the predicted depth map and the ground-truth depth map for the given training sample,
• wherein the training iteration is repeated to optimise the loss function to determine parameter values for the neural network architecture.