# US 12,602,898 B2 · Neural semantic fields for generalizable semantic segmentation of 3D scenes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,898 B2 - Document ID: US12602898B2 - Status: Granted - Kind: B2 - Application number: 18/012,264 - Filed: 2022-10-10 - Published: 2024-04-11 - Granted: 2026-04-14 - Earliest claimed: 2021-11-16 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Daniel Christopher Duckworth; Suhani Deepak-Ranu Vora; Noha Radwan; Klaus Greff; Henning Meyer; Kyle Adam Genova; Seyed Mohammad Mehdi Sajjadi; Etienne François Régis Pot; Andrea Tagliasacchi - Web page: https://patentgrep.com/patent/US12602898B2 ## CPC - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06T 7/11: Region-based segmentation - G06T 7/143: involving probabilistic approaches, e.g. Markov random field [MRF] modelling - G06T 7/194: involving foreground-background segmentation - G06T 15/08: Volume rendering - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06V 10/82: using neural networks - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Example embodiments of the present disclosure provide an example computer-implemented method for constructing a three-dimensional semantic segmentation of a scene from two-dimensional inputs. The example method includes obtaining, by a computing system comprising one or more processors, an image set comprising one or more views of a subject scene. The example method includes generating, by the computing system and based at least in part on the image set, a scene representation describing the subject scene in three dimensions. The example method includes generating, by the computing system and using a machine-learned semantic segmentation model framework, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over the three dimensions of the subject scene. The example method includes outputting, by the computing system, classification data for at least one location in the subject scene. ## Claims ### Claim 1 A computer-implemented method for constructing a three-dimensional semantic segmentation of a scene from two-dimensional inputs, comprising: generating, by a computing system comprising one or more processors and based on evaluating a first machine-learned model over a multidimensional lattice, a scene representation describing a subject scene in three dimensions; generating, by the computing system and using one or more second machine-learned models of a machine-learned semantic segmentation-model framework to process the scene representation, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over the three dimensions of the subject scene, the one or more second machine-learned models comprising segmentation weights learned based on a plurality of training example scenes; and outputting, by the computing system and based on the multidimensional field, classification data corresponding to one or more of the semantic categories for at least one location in the subject scene. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the first machine-learned model comprises scene-specific weights learned based on an image set comprising one or more views of the subject scene. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the first machine-learned model comprises a machine-learned density model, and wherein the scene representation provides a discretized representation of a density field across the subject scene. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, comprising: processing, by the computing system and using a geometric reasoning model of the one or more second machine-learned models, the scene representation to generate a feature lattice; interpolating, by the computing system, the feature lattice to obtain feature data corresponding to one or more locations in the subject scene; and decoding, by the computing system and using a machine-learned decoder model of the one or more second machine-learned models, the feature data to obtain one or more values of the multidimensional field at the one or more locations in the subject scene. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, comprising: outputting, by the computing system and using a volumetric rendering, a two-dimensional semantic map of the subject scene. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, comprising: performing, by the computing system, volumetric rendering over the multidimensional field along a ray from a viewpoint to the at least one location, the ray comprising the one or more locations. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, comprising: generating, by the computing system and based on evaluating a third machine-learned model over a second multidimensional lattice, a second scene representation describing a second subject scene in three dimensions, the third machine-learned model comprising scene-specific weights learned based on a second image set comprising one or more views of the second subject scene; generating, by the computing system and using the one or more second machine-learned models of the machine-learned semantic segmentation model framework to process the second scene representation, a second multidimensional field of probability distributions over semantic categories, the second multidimensional field defined over the three dimensions of the subject scene; and outputting, by the computing system and based on the second multidimensional field, second classification data corresponding to one or more of the semantic categories for at least one second location in the second subject scene. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the one or more second machine-learned models were trained without three-dimensional supervision. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the one or more second machine-learned models were trained using a segmentation loss determined using a reference two-dimensional segmentation map and a generated two-dimensional segmentation map, the generated two-dimensional segmentation map obtained by a volumetric rendering of a semantic feature field generated by the machine-learned semantic segmentation model framework. ### Claim 10 A computer-implemented method for training a machine-learned semantic segmentation model framework for three-dimensional semantic segmentation of novel scenes, comprising: training a set of segmentation parameters of the machine-learned semantic segmentation model framework over a plurality of training example scenes, the training comprising: obtaining, by a computing system comprising one or more processors, a reference map set comprising one or more two-dimensional reference semantic maps respectively associated with one or more views of a reference scene of the plurality of training example scenes; generating, by the computing system and the machine-learned semantic segmentation model framework, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over three dimensions of the reference scene; and updating, by the computing system, the set of segmentation parameters of the machine-learned semantic segmentation model framework based at least in part on comparing: the reference map set; and a rendered map set comprising one or more rendered semantic maps obtained from the multidimensional field respectively for the one or more views, the rendered map set obtained using the set of segmentation parameters. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein the machine-learned semantic segmentation model framework comprises a first machine-learned model configured to generate the multidimensional field based at least in part on a scene representation describing the reference scene in three dimensions. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein the machine-learned semantic segmentation model framework comprises a second machine-learned model configured to generate the scene representation, the second machine-learned model comprising scene-specific weights learned based on an image set comprising one or more unsegmented views of the reference scene. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the second machine-learned model is trained independently from the first machine-learned model. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, comprising: generating, by the computing system, a rendered semantic map based on: generating, by the computing system, based on evaluating a first machine-learned model over a multidimensional lattice, a scene representation describing the reference scene in three dimensions; generating, by the computing system and using one or more second machine-learned models of the machine-learned semantic segmentation model framework to process the scene representation, the multidimensional field; and performing, by the computing system, volumetric rendering over the multidimensional field along a ray from a viewpoint to a location in the reference scene. ### Claim 15 (depends on 12) The computer-implemented method of claim 12, comprising: processing, by the computing system and using a geometric reasoning model of the machine-learned semantic segmentation model framework, the scene representation to generate a feature lattice; interpolating, by the computing system, the feature lattice to obtain feature data corresponding to one or more locations in the reference scene; and decoding, by the computing system and using a machine-learned decoder model of machine-learned semantic segmentation model framework, the feature data to obtain one or more values of the multidimensional field at the one or more locations in the reference scene. ### Claim 16 A computing system for three-dimensional semantic segmentation of a scene, comprising: one or more processors; and one or more non-transitory computer-readable storage media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising: generating, based on evaluating a first machine-learned model over a multidimensional lattice, a scene representation describing a subject scene in three dimensions; generating, using one or more second machine-learned models of a machine-learned semantic segmentation-model framework to process the scene representation, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over the three dimensions of the subject scene, the one or more second machine-learned models comprising segmentation weights learned based on a plurality of training example scenes; and outputting, based on the multidimensional field, classification data corresponding to one or more of the semantic categories for at least one location in the subject scene. ### Claim 17 (depends on 16) The computing system of claim 16, wherein the first machine-learned model comprises scene-specific weights learned based on an image set comprising one or more views of the subject scene. ### Claim 18 (depends on 16) The computing system of claim 16, the operations comprising: processing, using a geometric reasoning model of the one or more second machine-learned models, the scene representation to generate a feature lattice; interpolating the feature lattice to obtain feature data corresponding to one or more locations in the subject scene; and decoding, using a machine-learned decoder model of the one or more second machine-learned models, the feature data to obtain one or more values of the multidimensional field at the one or more locations in the subject scene. ### Claim 19 (depends on 18) The computing system of claim 18, the operations comprising: performing volumetric rendering over the multidimensional field along a ray from a viewpoint to the at least one location, the ray comprising the one or more locations. ### Claim 20 (depends on 16) The computing system of claim 16, wherein the operations comprise: obtaining a two-dimensional reference semantic map associated with the at least one location in the subject scene; updating a set of segmentation parameters of the machine-learned semantic segmentation model framework based at least in part on comparing: the classification data; and the two-dimensional reference semantic map.