# US 2024/0391097 A1 · Systems and Methods for Sensor Data Packet Processing and Spatial Memory Updating for Robotic Platforms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0391097 A1
- Document ID: US20240391097A1
- Status: Published application
- Kind: A1
- Application number: 18/670,288
- Filed: 2024-05-21
- Published: 2024-11-28
- Earliest claimed: 2020-07-29
- Assignees: Aurora Operations, Inc.
- Applicants: Aurora Operations, Inc.
- Inventors: Sergio Casas; Davi Eugenio Nascimento Frossard; Shun Da Suo; Xuanyuan Tu; Raquel Urtasun
- Web page: https://patentgrep.com/patent/US20240391097A1

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- G01S 17/89: for mapping or imaging
- G06F 18/253: {of extracted features}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06N 20/00: Machine learning
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for streaming sensor packets in real-time are provided. An example method includes obtaining a sensor data packet representing a first portion of a three-hundred and sixty degree view of a surrounding environment of a robotic platform. The method includes generating, using machine-learned model(s), a local feature map based at least in part on the sensor data packet. The local feature map is indicative of local feature(s) associated with the first portion of the three-hundred and sixty degree view. The method includes updating, based at least in part on the local feature map, a spatial map to include the local feature(s). The spatial map includes previously extracted local features associated with a previous sensor data packet representing a different portion of the three-hundred and sixty degree view than the first portion. The method includes determining an object within the surrounding environment based on the updated spatial map.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A computer-implemented method, comprising:
obtaining, from a sensor system, a stream of sensor data packets representing respective angular slices of a view of a surrounding environment of an autonomous vehicle, wherein the stream of sensor data packets includes a first sensor data packet corresponding to a first angular slice of the view that is obtained at a first time when the first sensor data packet is acquired through the sensor system;
generating a first local feature map based at least in part on the first sensor data packet, wherein the first local feature map is indicative of one or more first local features associated with the first angular slice of the view;
updating a spatial map to include the first local features, wherein the spatial map comprises previously extracted local features associated with a previous sensor data packet representing a different angular slice of the view of the surrounding environment than the first angular slice;
determining an object within the surrounding environment of the autonomous vehicle based at least in part on the spatial map including the first local features; and
controlling motion of the autonomous vehicle based at least in part on the object determined within the surrounding environment.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein the stream of sensor data packets are received from the sensor system and iteratively processed in a streaming fashion as respective angular slices corresponding to a three-hundred and sixty degree view are obtained and updated in the spatial map.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, wherein the sensor system comprises one or more LIDAR sensors.

### Claim 24 (depends on 23)

The computer-implemented method of claim 23, wherein the sensor system ingests packets of LIDAR data from the one or more LIDAR sensors and emits the stream of sensor data packets without waiting for a full sweep to be built by the one or more LIDAR sensors.

### Claim 25 (depends on 23)

The computer-implemented method of claim 23, wherein the first sensor data packet comprises three-dimensional point cloud data generated by the one or more LIDAR sensors, wherein the one or more LIDAR sensors are rotated to generate sensor data packets for each respective angular slice of the view.

### Claim 26 (depends on 21)

The computer-implemented method of claim 21, wherein the stream of sensor data packets further includes a second sensor data packet corresponding to a second angular slice of the view that is obtained at a second time when the second sensor data packet is acquired through the sensor system after the first sensor data packet.

### Claim 27 (depends on 26)

The computer-implemented method of claim 26, further comprising:
generating a second local feature map based at least in part on the second sensor data packet, wherein the second local feature map is indicative of one or more second local features associated with the second angular slice of the view of the surrounding environment; and
updating, based at least in part on the second local feature map, the spatial map to include the one or more second local features;
wherein the second local feature map is generated after the first local feature map is generated.

### Claim 28 (depends on 21)

The computer-implemented method of claim 21, wherein generating the first local feature map comprises generating a two-dimensional representation associated with the first angular slice of the view of the surrounding environment of the autonomous vehicle.

### Claim 29 (depends on 21)

The computer-implemented method of claim 21, wherein determining the object comprises obtaining map data and fusing the map data with the spatial map.

### Claim 30 (depends on 21)

The computer-implemented method of claim 21, wherein updating the spatial map comprises replacing one or more previous computations of the spatial map with the one or more first local features.

### Claim 31 (depends on 21)

The computer-implemented method of claim 21, wherein updating the spatial map comprises:
obtaining previous computations descriptive of past computations for previous sensor data packets; and
updating spatial memory based on the previous sensor data packets and the first local feature map to update the spatial map.

### Claim 32 (depends on 21)

The computer-implemented method of claim 21, wherein the spatial map is descriptive of past local feature maps generated by processing previously obtained sensor data packets.

### Claim 33

A computing system comprising:
one or more processors; and
one or more computer-readable medium storing instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining, from a sensor system, a stream of sensor data packets representing respective angular slices of a view of a surrounding environment of an autonomous vehicle, wherein the stream of sensor data packets includes a first sensor data packet corresponding to a first angular slice of the view that is obtained at a first time when the first sensor data packet is acquired through the sensor system;
generating a first local feature map based at least in part on the first sensor data packet, wherein the first local feature map is indicative of one or more first local features associated with the first angular slice of the view;
updating a spatial map to include the first local features, wherein the spatial map comprises previously extracted local features associated with a previous sensor data packet representing a different angular slice of the view of the surrounding environment than the first angular slice;
determining an object within the surrounding environment of the autonomous vehicle based at least in part on the spatial map including the first local features; and
controlling motion of the autonomous vehicle based at least in part on the object determined within the surrounding environment.

### Claim 34 (depends on 33)

The computing system of claim 33, wherein the stream of sensor data packets are received from the sensor system and iteratively processed in a streaming fashion as respective angular slices corresponding to a three-hundred and sixty degree view are obtained and updated in the spatial map.

### Claim 35 (depends on 33)

The computing system of claim 33, wherein:
the sensor system comprises one or more LIDAR sensors; and
the sensor system ingests packets of LIDAR data from the one or more LIDAR sensors and emits the stream of sensor data packets without waiting for a full sweep to be built by the one or more LIDAR sensors.

### Claim 36 (depends on 35)

The computing system of claim 35, wherein the first sensor data packet comprises three-dimensional point cloud data generated by the one or more LIDAR sensors, wherein the one or more LIDAR sensors are rotated to generate sensor data packets for each respective angular slice of the view.

### Claim 37 (depends on 33)

The computing system of claim 33, wherein the stream of sensor data packets further includes a second sensor data packet corresponding to a second angular slice of the view that is obtained at a second time when the second sensor data packet is acquired through the sensor system after the first sensor data packet.

### Claim 38 (depends on 37)

The computing system of claim 37, the operations further comprising:
generating a second local feature map based at least in part on the second sensor data packet, wherein the second local feature map is indicative of one or more second local features associated with the second angular slice of the view of the surrounding environment; and
updating, based at least in part on the second local feature map, the spatial map to include the one or more second local features;
wherein the second local feature map is generated after the first local feature map is generated.

### Claim 39

An autonomous vehicle comprising:
a sensor system comprising one or more sensors;
one or more processors; and
one or more computer-readable medium storing instructions for execution by the one or more processors to cause the autonomous vehicle to perform operations, the operations comprising:
obtaining, from the sensor system, a stream of sensor data packets representing respective angular slices of a view of a surrounding environment of an autonomous vehicle, wherein the stream of sensor data packets includes a first sensor data packet corresponding to a first angular slice of the view that is obtained at a first time when the first sensor data packet is acquired through the sensor system;
generating a first local feature map based at least in part on the first sensor data packet, wherein the first local feature map is indicative of one or more first local features associated with the first angular slice of the view;
updating a spatial map to include the first local features, wherein the spatial map comprises previously extracted local features associated with a previous sensor data packet representing a different angular slice of the view of the surrounding environment than the first angular slice;
determining an object within the surrounding environment of the autonomous vehicle based at least in part on the spatial map including the first local features; and
controlling motion of the autonomous vehicle based at least in part on the object determined within the surrounding environment.

### Claim 40 (depends on 39)

The autonomous vehicle of claim 39, wherein the stream of sensor data packets are received from the sensor system and iteratively processed in a streaming fashion as respective angular slices corresponding to a three-hundred and sixty degree view are obtained and updated in the spatial map.
