# US 11,908,203 B2 · Analysis of point cloud data using depth maps > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,203 B2 - Document ID: US11908203B2 - Status: Granted - Kind: B2 - Application number: 17/718,721 - Filed: 2022-04-12 - Published: 2022-07-28 - Granted: 2024-02-20 - Earliest claimed: 2018-02-27 - Assignees: NVIDIA Corporation - Applicants: Nvidia Corporation - Inventors: Ishwar Kulkarni; Ibrahim Eden; Michael Kroepfl; David Nister - Web page: https://patentgrep.com/patent/US11908203B2 ## CPC - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06T 15/04: Texture mapping - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/582: {of traffic signs} - G06V 20/584: {of vehicle lights or traffic lights} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30241: Trajectory - G06V 2201/12: Acquisition of 3D measurements of objects - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. Improved techniques for processing the point cloud data that has been collected are provided. The improved techniques include mapping one or more point cloud data points into a depth map, the one or more point cloud data points being generated using one or more sensors; determining one or more mapped point cloud data points within a bounded area of the depth map, and detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities based on the one or more mapped point cloud data points. ## Claims ### Claim 1 A method comprising: mapping one or more motion-compensated point cloud data points into a depth map, the one or more point cloud data points being generated using one or more sensors; determining one or more mapped point cloud data points within a bounded area of the depth map; and detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities based on the one or more mapped point cloud data points. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more entities comprises at least one of: a location of the machine, an object in the environment, a traversable freespace in the environment, or a landmark in the environment. ### Claim 3 (depends on 1) The method of claim 1, wherein said mapping comprises transforming the one or more point cloud data points into a polar depth map that represents one or more distances between the one or more point cloud data points and the one or more sensors. ### Claim 4 (depends on 3) The method as recited in claim 3, further comprising: determining plane information for the data points from the polar depth map; and storing the plane information and the polar depth map using a two dimensional (2D) depth map, wherein said storing includes transforming the data points in the polar depth map into Euclidean coordinates using nearest neighbor analysis. ### Claim 5 (depends on 4) The method as recited in claim 4, wherein said storing further includes computing a depth value for the one or more of the point cloud data points by intersecting a nominal ray with a plane corresponding to the one or more point cloud data points, and storing the depth values in the 2D depth map. ### Claim 6 (depends on 4) The method as recited in claim 4, wherein said storing further includes using a texture map and analyzing the texture map using processing constructs of the one or more processing units. ### Claim 7 (depends on 4) The method as recited in claim 4, wherein said determining plane information utilizes a best fit model over a bounded window of the point cloud data points. ### Claim 8 (depends on 7) The method as recited in claim 7, wherein the bounded window defines a block of the point cloud data points, on which a nearest neighbor analysis is performed. ### Claim 9 (depends on 7) The method as recited in claim 7, wherein the bounded window utilizes a fixed grid of the point cloud data points. ### Claim 10 (depends on 1) The method as recited in claim 1, further comprising: using the one or more sensors to generate, at a periodic interval, multiple slices of the point cloud data points. ### Claim 11 (depends on 10) The method as recited in claim 10, further comprising: estimating one or more trajectory equations using nearest neighbor analysis and the multiple slices of the point cloud data points. ### Claim 12 (depends on 11) The method as recited in claim 11, further comprising: calculating a movement of the object utilizing the one or more trajectory equations, wherein said calculating includes identifying a future position of the object using at least one of the one or more trajectory equations that corresponds to the object. ### Claim 13 (depends on 11) The method as recited in claim 11, further comprising: calculating a trajectory of the machine utilizing the one or more trajectory equations, wherein said calculating includes identifying motion vectors for the machine and surrounding objects from the multiple slices of the point cloud data points and aggregating identified motion vectors. ### Claim 14 (depends on 11) The method as recited in claim 11, further comprising: calculating a trajectory of the machine by producing transformations among the slices of the point cloud data points and inversing the transformations. ### Claim 15 (depends on 10) The method as recited in claim 10, further comprising: storing each of the slices as a portion of the depth map; and using, for each of the slices, a first angle parameter for a horizontal dimension, and a second angle parameter for a vertical dimension. ### Claim 16 (depends on 10) The method as recited in claim 10, further comprising: estimating one or more trajectory equations by matching some of the mapped point cloud data points in the slices that correspond to a same object and calculating a distance between the some of the matched data points. ### Claim 17 (depends on 10) The method as recited in claim 10, further comprising: varying a size of the bounded area of the depth map based an angular separation between the slices. ### Claim 18 (depends on 10) The method as recited in claim 10, further comprising: calculating a movement of the object and a trajectory of the machine and determine an action that would prevent the object from coming into contact with the machine based on the movement of the object or the trajectory of the machine. ### Claim 19 (depends on 1) The method as recited in claim 1, wherein the one or more sensors are part of a LiDAR system, a RADAR system or a camera system. ### Claim 20 (depends on 1) The method as recited in claim 1, wherein said mapping comprises transforming the one or more point cloud data points to a two dimensional (2D) depth map. ### Claim 21 (depends on 1) The method as recited in claim 1, further comprising: determining a neighbor data point that is nearest to each of the one or more mapped point cloud data points within the bounded area. ### Claim 22 (depends on 1) The method as recited in claim 1, wherein said mapping includes recording a single depth value for a set of the mapped point cloud data points that represent a region of the depth map. ### Claim 23 (depends on 22) The method as recited in claim 22, wherein the single depth value is an average depth value of the region or the minimum depth value in the region. ### Claim 24 A system comprising: one or more processing units to perform one or more operations, the one or more operations including: obtaining one or more data points of a point cloud, the one or more point cloud data points being generated using one or more sensors; generating, for at least one data point of the one or more point cloud data points, at least one motion-compensated data point by computing a motion compensation for the at least one data point based at least on a motion of at least one sensor of the one or more sensors; mapping the one or more point cloud data points, including the at least one motion-compensated data point, into a depth map; determining one or more mapped point cloud data points within a bounded area of the depth map; and detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities in the environment based on the one or more mapped point cloud data points. ### Claim 25 (depends on 24) The system as recited in claim 24, further comprising using the one or more sensors to generate multiple slices of the one or more point cloud data points. ### Claim 26 A processor comprising: one or more processing units to associate one or more motion-compensated data points of a point cloud to a bounded area of a depth map, and to detect a location of one or more entities in an environment using the associated one or more motion-compensated data points, wherein the one or more motion-compensated data points are obtained from one or more sensors observing the environment.