# US 2019/0377965 A1 · Systems and Methods for Pipelined Processing of Sensor Data Using Hardware

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377965 A1
- Document ID: US20190377965A1
- Status: Published application
- Kind: A1
- Application number: 16/131,529
- Filed: 2018-09-14
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: Uber Technologies, Inc.
- Applicants: Uber Technologies, Inc.
- Inventors: George Totolos, JR.; Joshua Oren Silberman
- Web page: https://patentgrep.com/patent/US20190377965A1

## CPC

- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 18/24: Classification techniques
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N 3/063: using electronic means
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one example embodiment, a computer-implemented method includes obtaining sensor data from a sensor, the sensor data corresponding to an image frame, and the sensor data including a first portion that corresponds to a portion of the image frame. The method includes pipelining the first portion of the sensor data into a machine-learned model before the sensor data corresponding to the entire image frame is transferred from the sensor to a memory device, to perform one or more inference operations on the first portion of the sensor data. The method includes generating as an output of the machine-learned model, in response to pipelining the sensor data corresponding to each portion of the image frame into the machine-learned model, a detection or classification of the one or more objects indicated within the sensor data.

## Claims

### Claim 1

An image processing system, comprising:
a field programmable gate array (FPGA) device coupled to one or more sensors that are configured to obtain sensor data corresponding to an image frame, the FPGA device comprising one or more memory devices that are configured to store a representation of a machine-learned model that detects or classifies one or more objects indicated within the sensor data, the FPGA device further configured to receive a plurality of increments of the sensor data as each increment of the sensor data becomes available and to implement one or more image processing pipelines for pre-processing the sensor data,
wherein the one or more image processing pipelines include a first plurality of logic blocks and interconnectors programmed to extract one or more features from one or more increments of the plurality of increments of the sensor data and generate first input data, each increment of the sensor data corresponding to a portion of the image frame, and
wherein the first input data is input as one or more increments to one or more layers of the machine-learned model, the machine-learned model processing each increment of the first input data as the first input data is received at the one or more layers, to provide an output indicative of a detection or classification of the one or more objects indicated within the sensor data.

### Claim 2 (depends on 1)

The image processing system of claim 1, wherein the FPGA device receives the sensor data in the one or more increments and the FPGA pipelines the one or more increments of the sensor data into the one or more image processing pipelines as each increment of the sensor data is received.

### Claim 3 (depends on 1)

The image processing system of claim 1, wherein the first plurality of logic blocks and interconnectors are programmed to provide the first input data to a second plurality of logic blocks and interconnectors programmed to receive the first input data and generate second input data as each increment of the first input data is output by the first plurality of logic blocks and interconnectors, and
wherein the second input data is input as one or more increments to one or more layers of the machine-learned model, the machine-learned model processing each increment of the second input data as the second input data is received at the one or more layers, to provide the output indicative of the detection or classification of the one or more objects indicated within the sensor data.

### Claim 4 (depends on 1)

The image processing system of claim 1, wherein the FPGA includes a third plurality of logic blocks and interconnectors programmed to implement the machine-learned model, and to provide the output indicative of the detection or classification of the one or more objects indicated within the sensor data in response to receiving the first input data and second input data that corresponds to the entire image frame.

### Claim 5 (depends on 1)

The image processing system of claim 1, wherein the first plurality of logic blocks and interconnectors are programmed to extract the one or more features from each increment of the sensor data in parallel.

### Claim 6 (depends on 1)

The image processing system of claim 1, wherein the first plurality of logic blocks and interconnectors are programmed to extract the one or more features associated with a state of the one or more objects indicated within the sensor data.

### Claim 7 (depends on 1)

The image processing system of claim 1, wherein the sensor data is associated with a surrounding environment of the autonomous vehicle, and wherein the one or more objects are located within the surrounding environment of the autonomous vehicle.

### Claim 8 (depends on 1)

The image processing system of claim 1, wherein the sensor data is image data and each increment of data corresponds to a row of pixels in the image data.

### Claim 9 (depends on 1)

The image processing system of claim 1, wherein the sensor data is LIDAR data that represents a 360 degree scene of an environment, and each increment of the sensor data corresponds to an angular slice of the 360 degree scene.

### Claim 10

An autonomous vehicle, comprising:
a field programmable gate array (FPGA) device coupled to one or more sensors that are configured to obtain sensor data corresponding to an image frame, the FPGA device comprising one or more memory devices that are configured to store a representation of a machine-learned model that detects or classifies one or more objects indicated within the sensor data, the FPGA device further configured to receive a plurality of increments of the sensor data as each increment of the sensor data becomes available and to implement one or more image processing pipelines for pre-processing the sensor data,
wherein the one or more image processing pipelines include a first plurality of logic blocks and interconnectors programmed to extract one or more features from one or more increments of the plurality of increments of the sensor data and generate first input data, each increment of the sensor data corresponding to a portion of the image frame, and
wherein the first input data is input as one or more increments to one or more layers of the machine-learned model, the machine-learned model processing each increment of the first input data as the first input data is received at the one or more layers, to provide an output indicative of a detection or classification of the one or more objects indicated within the sensor data.

### Claim 11 (depends on 10)

The autonomous vehicle of claim 10, wherein the FPGA device receives the sensor data in the one or more increments and the FPGA pipelines the one or more increments of the sensor data into the one or more image processing pipelines as each increment of the sensor data is received.

### Claim 12 (depends on 10)

The autonomous vehicle of claim 10, wherein the first plurality of logic blocks and interconnectors are programmed to provide the first input data to a second plurality of logic blocks and interconnectors programmed to receive the first input data and generate second input data as each increment of the first input data is output by the first plurality of logic blocks and interconnectors, and
wherein the second input data is input as one or more increments to one or more layers of the machine-learned model, the machine-learned model processing each increment of the second input data as the second input data is received at the one or more layers, to provide the output indicative of the detection or classification of the one or more objects indicated within the sensor data.

### Claim 13 (depends on 10)

The autonomous vehicle of claim 10, wherein the FPGA includes a third plurality of logic blocks and interconnectors programmed to implement the machine-learned model, and to provide the output indicative of the detection or classification of the one or more objects indicated within the sensor data in response to receiving the first input data and second input data that corresponds to the entire image frame.

### Claim 14

A computer-implemented method comprising:
obtaining, by a computing system that includes one or more computing devices, sensor data that is outputted by one or more sensors as the sensor data becomes available, wherein the sensor data includes a plurality of increments of data, the computing system receiving each of the plurality of increments of data;
determining, by the computing system, a number of increments of data associated with a first inference operation;
inputting, by the computing system into a machine-learned model, a first portion of the sensor data upon receipt of the first portion of sensor data, wherein the first portion of the sensor data comprises a subset of the plurality of increments of data that corresponds to the number of increments associated with the first inference operation, and the machine-learned model causes the first inference operation to be performed on the first portion of the sensor data; and
obtaining, by the computing system as an output of the machine-learned model based at least in part on the first inference operation, a classification of one or more objects indicated within the sensor data.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, further comprising:
inputting, by the computing system, a second portion of the sensor data into the machine-learned model, wherein the second portion of the sensor data corresponds to the number of increments of data associated with the first inference operation, and the second portion of the sensor data includes at least one increment of data included in the first portion of the sensor data and at least one increment of data that was not included in the first portion of the sensor data.

### Claim 16 (depends on 14)

The computer-implemented method of claim 14, further comprising:
determining, by the computing system, a number of increments associated with a second inference operation;
inputting, by the computing system into the machine-learned model, a third portion of the sensor data upon receipt of the third portion of sensor data, wherein the third portion of the sensor data corresponds to the number of increments of data associated with the second inference operation, and the machine-learned model causes the second inference operation to be performed on the third portion of the sensor data.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein the first portion of the sensor data and the third portion of the sensor data are simultaneously input into the machine-learned model.

### Claim 18 (depends on 14)

The computer-implemented method of claim 14, wherein the computing system comprises a field programmable gate array configured to perform the first inference operation, and wherein the one or more sensors are configured to communicate directly with the field programmable gate array.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the sensor and the field programmable gate array are installed on a single chip.

### Claim 20 (depends on 14)

The computer-implemented method of claim 14, wherein the sensor data corresponds to an environment of an autonomous vehicle, and the method further comprises:
controlling, by the computing system, the autonomous vehicle to navigate the environment based at least in part on the classification of the one or more objects indicated within the sensor data.
