Neural semantic fields for generalizable semantic segmentation of 3D scenes

A computer-implemented method for training a machine-learned semantic segmentation model framework for three-dimensional semantic segmentation of novel scenes, comprising:

• training a set of segmentation parameters of the machine-learned semantic segmentation model framework over a plurality of training example scenes, the training comprising:
• obtaining, by a computing system comprising one or more processors, a reference map set comprising one or more two-dimensional reference semantic maps respectively associated with one or more views of a reference scene of the plurality of training example scenes;
• generating, by the computing system and the machine-learned semantic segmentation model framework, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over three dimensions of the reference scene;
• and
• updating, by the computing system, the set of segmentation parameters of the machine-learned semantic segmentation model framework based at least in part on comparing:
• the reference map set;
• and
• a rendered map set comprising one or more rendered semantic maps obtained from the multidimensional field respectively for the one or more views, the rendered map set obtained using the set of segmentation parameters.

A computing system for three-dimensional semantic segmentation of a scene, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable storage media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
• generating, based on evaluating a first machine-learned model over a multidimensional lattice, a scene representation describing a subject scene in three dimensions;
• generating, using one or more second machine-learned models of a machine-learned semantic segmentation-model framework to process the scene representation, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over the three dimensions of the subject scene, the one or more second machine-learned models comprising segmentation weights learned based on a plurality of training example scenes;
• and
• outputting, based on the multidimensional field, classification data corresponding to one or more of the semantic categories for at least one location in the subject scene.