# US 2022/0036650 A1 · Systems and methods for multi-resolution fusion of pseudo-lidar features > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036650 A1 - Document ID: US20220036650A1 - Status: Published application - Kind: A1 - Application number: 16/940,701 - Filed: 2020-07-28 - Published: 2022-02-03 - Earliest claimed: 2020-07-28 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Victor Vaquero Gomez; Rares A. Ambrus; Vitor Guizilini; Adrien D. Gaidon - Web page: https://patentgrep.com/patent/US20220036650A1 ## CPC - G06T 17/05: Geographic models - G06V 10/82: using neural networks - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 30/1912: {Selecting the most significant subset of features (G06V30/19127 takes precedence)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The embodiments disclosed herein describe vehicles, systems and methods for multi-resolution fusion of pseudo-LiDAR features. In one aspect, a method for multi-resolution fusion of pseudo-LiDAR features includes receiving image data from one or more image sensors, generating a point cloud from the image data, generating, from the point cloud, a first bird's eye view map having a first resolution, generating, from the point cloud, a second bird's eye view map having a second resolution, and generating a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map. ## Claims ### Claim 1 A method for multi-resolution fusion of pseudo-LiDAR features, the method comprising: receiving image data from one or more image sensors; generating a point cloud from the image data, wherein the point cloud comprises pseudo-LiDAR data; generating, from the point cloud, a first bird's eye view map having a first resolution; generating, from the point cloud, a second bird's eye view map having a second resolution; and generating a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating, from the point cloud, one or more additional bird's eye view maps having one or more additional resolutions, wherein generating the combined bird's eye view map further comprises combining features of the one or more additional bird's eye view maps with the features of the first bird's eye view map and the second bird's eye view map. ### Claim 3 (depends on 1) The method of claim 1, wherein: generating the first bird's eye view map comprises: subdividing the pseudo-LiDAR data into a plurality of voxels having a first volume that is determined by the first resolution; and generating an array of first cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data; generating the second bird's eye view map comprises: subdividing the pseudo-LiDAR data into a plurality of voxels having a second volume that is determined by the first resolution; and generating an array of second cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data. ### Claim 4 (depends on 3) The method of claim 3, wherein the generating of the combined bird's eye view map comprises combining the one or more features from each first cell with the one or more features from each second cell that align with first cells. ### Claim 5 (depends on 1) The method of claim 1, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by an addition function. ### Claim 6 (depends on 1) The method of claim 1, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by a neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by concatenation. ### Claim 8 A method of detecting an object, the method comprising: generating a bird's eye view of pseudo-LiDAR data by: receiving image data from one or more image sensors; generating a point cloud from the image data, wherein the point cloud comprises pseudo-LiDAR data; generating, from the point cloud, a first bird's eye view map having a first resolution; generating, from the point cloud, a second bird's eye view map having a second resolution; and generating a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map; and detecting, using an object detection algorithm, one or more objects from the combined bird's eye view map. ### Claim 9 (depends on 8) The method of claim 8, further comprising generating, from the point cloud, one or more additional bird's eye view maps having one or more additional resolutions, wherein generating the combined bird's eye view map further comprises combining features of the one or more additional bird's eye view maps with the features of the first bird's eye view map and the second bird's eye view map. ### Claim 10 (depends on 8) The method of claim 8, wherein: generating the first bird's eye view map comprises: subdividing the pseudo-LiDAR data into a plurality of voxels having a first volume that is determined by the first resolution; and generating an array of first cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data; generating the second bird's eye view map comprises: subdividing the pseudo-LiDAR data into a plurality of voxels having a second volume that is determined by the first resolution; and generating an array of second cells comprising one or more layers by extracting one or more features from the pseudo-LiDAR data within the plurality of voxels, wherein each layer of the one or more layers comprises a plurality of pixels, each pixel comprises an individual feature extracted from an individual voxel, and a number of layers equals a number of features extracted from the pseudo-LiDAR data. ### Claim 11 (depends on 10) The method of claim 10, wherein the generating of the combined bird's eye view map comprises combining the one or more features from each first cell with the one or more features from each second cell that align with first cells in a vertical direction. ### Claim 12 (depends on 8) The method of claim 8, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by an addition function. ### Claim 13 (depends on 8) The method of claim 8, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by a neural network. ### Claim 14 (depends on 8) The method of claim 8, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by concatenation. ### Claim 15 (depends on 8) The method of claim 8, wherein the object detection algorithm comprises an object detection neural network. ### Claim 16 A vehicle comprising: one or more image sensors that produce image data; one or more processors; one or memory modules comprising a non-transitory computer readable memory storing computer readable instructions that, when executed by the one or more processors, cause the one or more processors to: receive the image data from the one or more image sensors; generate a point cloud from the image data, wherein the point cloud comprises pseudo-LiDAR data; generate, from the point cloud, a first bird's eye view map having a first resolution; generate, from the point cloud, a second bird's eye view map having a second resolution; generate a combined bird's eye view map by combining features of the first bird's eye view map with features from the second bird's eye view map; and detect, using an object detection algorithm, one or more objects from the combined bird's eye view map. ### Claim 17 (depends on 16) The vehicle of claim 16, wherein the computer readable instructions further cause the one or more processors to control a movement of the vehicle based at least in part on the detected one or more objects. ### Claim 18 (depends on 16) The vehicle of claim 16, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by an addition function. ### Claim 19 (depends on 16) The vehicle of claim 16, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by a neural network. ### Claim 20 (depends on 16) The vehicle of claim 16, wherein the features of the first bird's eye view map are combined with the features from the second bird's eye view map by concatenation.