# US 2021/0343022 A1 · Multi-modal sensor data association architecture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343022 A1
- Document ID: US20210343022A1
- Status: Published application
- Kind: A1
- Application number: 17/373,550
- Filed: 2021-07-12
- Published: 2021-11-04
- Earliest claimed: 2019-04-16
- Assignees: Zoox, Inc.
- Applicants: Zoox, Inc.
- Inventors: Joshua Kriser Cohen; Sabeek Mani Pradhan; Balazs Kovacs; Cooper Stokes Sloan
- Web page: https://patentgrep.com/patent/US20210343022A1

## CPC

- G01S 7/417: {involving the use of neural networks}
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/89: for mapping or imaging
- G01S 13/931: of land vehicles
- G01S 17/89: for mapping or imaging
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/20: Ensemble learning
- G06T 7/11: Region-based segmentation
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/931: of land vehicles
- G01S 2013/932: {using own vehicle data, e.g. ground speed, steering wheel direction}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 20/00: Machine learning
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 2201/12: Acquisition of 3D measurements of objects
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A machine-learning architecture may be trained to determine point cloud data associated with different types of sensors with an object detected in an image and/or generate a three-dimensional region of interest (ROI) associated with the object. In some examples, the point cloud data may be associated with sensors such as, for example, a lidar device, radar device, etc.

## Claims

### Claim 1

A method comprising:
receiving first sensor data associated with a first type of sensor, the first sensor data representing a portion of an environment surrounding an autonomous vehicle;
receiving second sensor data associated with a second type of sensor, the second sensor data representing a same portion or different portion of the environment as the portion represented by the first sensor data;
receiving an object detection, wherein the object detection identifies an object in one or more images;
determining, based at least in part on the object detection, a first subset of the first sensor data and a second subset of the second sensor data;
inputting the first subset of the first sensor data into a first subnetwork;
inputting the second subset of the second sensor data into a second subnetwork;
receiving a first output from the first subnetwork and a second output from the second subnetwork;
combining, as a combined output, the first output and the second output;
inputting a first portion of the combined output into a third subnetwork and a second portion of the combined output into a fourth subnetwork; and
receiving a first map from the third subnetwork and a second map from the fourth subnetwork, wherein:
the first map indicates at least a first probability that a first point of the first sensor data is associated with the object, and
the second map indicates at least a second probability that a second point of the second sensor data is associated with the object.
