# US 12,223,705 B2 · Semantic segmentation of three-dimensional data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,705 B2 - Document ID: US12223705B2 - Status: Granted - Kind: B2 - Application number: 18/299,970 - Filed: 2023-04-13 - Published: 2023-08-10 - Granted: 2025-02-11 - Earliest claimed: 2017-11-15 - Assignees: Aurora Operations, Inc. - Applicants: Aurora Operations, Inc. - Inventors: Chris Jia-Han Zhang; Wenjie Luo; Raquel Urtasun - Web page: https://patentgrep.com/patent/US12223705B2 ## CPC - G06V 10/82: using neural networks - G01S 7/4808: {Evaluating distance, position or velocity data} - G06T 3/06: Topological mapping of higher dimensional structures onto lower dimensional surfaces - G06V 10/764: using classification, e.g. of video objects - G06V 10/776: Validation; Performance evaluation - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/64: Three-dimensional [3D] objects - G01S 17/89: for mapping or imaging - G06V 2201/12: Acquisition of 3D measurements of objects - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods for performing semantic segmentation of three-dimensional data are provided. In one example embodiment, a computing system can be configured to obtain sensor data including three-dimensional data associated with an environment. The three-dimensional data can include a plurality of points and can be associated with one or more times. The computing system can be configured to determine data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data. The computing system can be configured to determine a classification for each point of the plurality of points within the three-dimensional data based at least in part on the two-dimensional voxel representation associated with the environment and a machine-learned semantic segmentation model. The computing system can be configured to initiate one or more actions based at least in part on the per-point classifications. ## Claims ### Claim 1 A computer-implemented method of semantic segmentation, the method comprising: obtaining sensor data comprising three-dimensional data associated with an environment; determining data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data and a voxel grid, wherein the two-dimensional voxel representation is associated with one or more voxels; determining a voxel classification for respective voxels of the one or more voxels associated with the two-dimensional voxel representation, based at least in part on the two-dimensional voxel representation; and determining a classification for respective points of a plurality of points within the three-dimensional data based at least in part on the voxel classification for the respective voxels. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein determining a voxel classification for a respective voxel comprises: determining that a voxel of the one or more voxels is associated with an object in the environment; and determining a classification of the object based on the voxel classification. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the classification is indicative of (i) a static object or (ii) a dynamic object. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: determining that the sensor data is associated with a plurality of times; and determining the classification for the respective points across the plurality of times. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the classification for the respective points across the plurality of times is associated with motion of one or more objects in the environment. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising determining a classification of one or more portions of the environment based on the voxel classification for the respective voxels. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: determining a probability distribution across a plurality of classes for the respective voxels; and predicting the voxel classification for the respective voxels based on the probability distribution. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the voxel grid comprises a plurality of voxel cells. ### Claim 9 A vehicle computing system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to perform operations, the operations comprising: obtaining sensor data comprising three-dimensional data associated with an environment; determining data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data and a voxel grid, wherein the two-dimensional voxel representation is associated with one or more voxels; determining a voxel classification for respective voxels of the one or more voxels associated with the two-dimensional voxel representation, based at least in part on the two-dimensional voxel representation; and determining a classification for respective points of a plurality of points within the three-dimensional data based at least in part on the voxel classification for the respective voxels. ### Claim 10 (depends on 9) The vehicle computing system of claim 9, wherein determining a voxel classification for a respective voxel comprises: determining that a voxel of the one or more voxels is associated with an object in the environment; and determining a classification of the object based on the voxel classification. ### Claim 11 (depends on 10) The vehicle computing system of claim 10, wherein the classification is indicative of (i) a static object or (ii) a dynamic object. ### Claim 12 (depends on 9) The vehicle computing system of claim 9, further comprising: determining that the sensor data is associated with a plurality of times; and determining the classification for the respective points across the plurality of times. ### Claim 13 (depends on 12) The vehicle computing system of claim 12, wherein the classification for the respective points across the plurality of times is associated with motion of one or more objects in the environment. ### Claim 14 (depends on 9) The vehicle computing system of claim 9, further comprising determining a classification of one or more portions of the environment based on the voxel classification for the respective voxels. ### Claim 15 (depends on 9) The vehicle computing system of claim 9, further comprising: determining a probability distribution across a plurality of classes for the respective voxels; and predicting the voxel classification for the respective voxels based on the probability distribution. ### Claim 16 (depends on 9) The vehicle computing system of claim 9, wherein the voxel grid comprises a plurality of voxel cells. ### Claim 17 One or more non-transitory computer-readable media storing instructions executable by one or more processors to cause the one or more processors to perform operations, the operations comprising: obtaining sensor data comprising three-dimensional data associated with an environment; determining data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data and a voxel grid, wherein the two-dimensional voxel representation is associated with one or more voxels; determining a voxel classification for respective voxels of the one or more voxels associated with the two-dimensional voxel representation, based at least in part on the two-dimensional voxel representation; and determining a classification for respective points of a plurality of points within the three-dimensional data based at least in part on the voxel classification for the respective voxels. ### Claim 18 (depends on 17) The one or more non-transitory computer-readable media of claim 17, wherein determining a voxel classification for a respective voxel comprises: determining that a voxel of the one or more voxels is associated with an object in the environment; and determining a classification of the object based on the voxel classification. ### Claim 19 (depends on 18) The one or more non-transitory computer-readable media of claim 18, wherein the classification is indicative of (i) a static object or (ii) a dynamic object. ### Claim 20 (depends on 17) The one or more non-transitory computer-readable media of claim 17, further comprising: determining that the sensor data is associated with a plurality of times; and determining the classification for the respective points across the plurality of times.