# US 2022/0284619 A1 · Offline optimization of sensor data for agent trajectories

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284619 A1
- Document ID: US20220284619A1
- Status: Published application
- Kind: A1
- Application number: 17/190,312
- Filed: 2021-03-02
- Published: 2022-09-08
- Earliest claimed: 2021-03-02
- Assignees: Lyft, Inc.
- Applicants: Lyft, Inc.
- Inventors: Filippo Brizzi; Luca del Pero; Tayyab Naseer; Lorenzo Peppoloni
- Web page: https://patentgrep.com/patent/US20220284619A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- B60W 10/18: including control of braking systems
- B60W 10/20: including control of steering systems
- B60W 30/00: Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W 60/001: {Planning or execution of driving tasks}
- B60W 2420/403: Image sensing, e.g. optical camera
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/30241: Trajectory
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Examples disclosed herein involve a computing system configured to (i) based on image data captured by a vehicle in an environment, obtain observations of a time-sequence of positions of an agent identified within the image data, (ii) generate a first updated time-sequence of positions of the agent by performing a first optimization operation that includes processing the observed time-sequence of positions by beginning with a position associated with an observation having the highest confidence in the time-sequence of observations and proceeding in a first direction, (iii) after generating the first updated time-sequence of positions, generate a second updated time-sequence of positions of the agent by performing a second optimization operation that includes processing the first updated time-sequence of positions in a second direction opposite the first direction, and (iv) derive a trajectory for the agent in the environment based on the second updated time-sequence of positions for the agent.

## Claims

### Claim 1

A computer-implemented method comprising:
based on image data captured by a vehicle in an environment, obtaining observations of a time-sequence of positions of an agent identified within the image data;
generating a first updated time-sequence of positions of the agent by performing a first optimization operation on the observed time-sequence of positions, wherein the first optimization operation includes processing at least two or more positions of the observed time-sequence of positions by beginning with a position associated with an observation having the highest confidence in the time-sequence of observations and proceeding in a first direction;
after generating the first updated time-sequence of positions, generating a second updated time-sequence of positions of the agent by performing a second optimization operation on the first updated time-sequence of positions, wherein the second optimization operation includes processing at least two or more positions of the first updated time-sequence of positions in a second direction opposite the first direction; and
deriving a trajectory for the agent in the environment based on the second updated time-sequence of positions for the agent.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein obtaining the observations of the time-sequence of positions of the agent comprises:
tracking observations of the agent identified across a series of image frames within the image data;
assigning a common agent identifier (ID) to each of the tracked observations of the agent; and
assembling the tracked observations comprising the common agent ID into the observations of the time-sequence of positions.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein tracking the observations of the agent identified across the series of image frames within the image data comprises:
based on (i) one or more respective prior positions associated with the one or more previous observations of the agent and (ii) a motion model for the agent, determining a predicted position of the agent; and
determining a match between the predicted position of the agent and a future observation of the agent based on:
an intersection of the predicted position of the agent and the future observation of the agent; and
a visual matching analysis of the predicted position of the agent with the future observation of the agent.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining a respective measurement error associated with each observation in the observations of the time-sequence of positions that is based on a measured distance of the positions from the vehicle; and
incorporating the respective error value associated with each observation into at least the first optimization operation or the second optimization operation.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the respective error value increases quadratically as the measured distance from the vehicle increases.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
augmenting the second updated time-sequence of observations to include one or more additional intermediate observations between each pair of observations in the second updated time-sequence of observations.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the image data is captured by a stereo camera having a frame rate of 5 frames per second, and wherein augmenting the second updated time-sequence of observations comprises increasing the frame rate of the second updated time-sequence of observations from 5 frames per second to 30 frames per second.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein beginning with the position associated with the observation having the highest confidence in the time-sequence of observations and proceeding in the first direction comprises:
determining that one of (i) an earliest position in the time-sequence of positions or (ii) a latest position in the time-sequence of position is the position of the agent associated with the observation having the highest confidence in the time-sequence of observations based on the measured distance of each respective position from the vehicle; and
proceeding (i) chronologically forward or (ii) chronologically backward, respectively.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein generating the first updated time-sequence of positions of the agent comprises processing at least two or more positions of the observed time-sequence of positions by beginning with a position associated with an earliest observation in the time-sequence of observations and proceeding chronologically forward; and
wherein generating the second updated time-sequence of positions of the agent comprises processing at least two or more positions of the first updated time-sequence of observations by beginning with a position associated with a latest observation and proceeding chronologically backward.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein performing the first optimization operation on the observed time-sequence of positions comprises:
based on (i) an initial observed position in the time-sequence of positions and (ii) a motion model for the agent, determining a next predicted position for the agent;
based on (i) a next observed position for the agent and (ii) the next predicted position for the agent, determining an updated next position for the agent having a higher confidence than the next observed position for the agent; and
using the updated next position for the agent as a next initial observed position in the time-sequence of positions to thereby determine the first updated time-sequence of positions.

### Claim 11

A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:
based on image data captured by a vehicle in an environment, obtain observations of a time-sequence of positions of an agent identified within the image data;
generate a first updated time-sequence of positions of the agent by performing a first optimization operation on the observed time-sequence of positions, wherein the first optimization operation includes processing at least two or more positions of the observed time-sequence of positions by beginning with a position associated with an observation having the highest confidence in the time-sequence of observations and proceeding in a first direction;
after generating the first updated time-sequence of positions, generate a second updated time-sequence of positions of the agent by performing a second optimization operation on the first updated time-sequence of positions, wherein the second optimization operation includes processing at least two or more positions of the first updated time-sequence of positions in a second direction opposite the first direction; and
derive a trajectory for the agent in the environment based on the second updated time-sequence of positions for the agent.

### Claim 12 (depends on 11)

The computer-readable medium of claim 11, wherein obtaining the observations of the time-sequence of positions of the agent comprises:
tracking observations of the agent identified across a series of image frames within the image data;
assigning a common agent identifier (ID) to each of the tracked observations of the agent; and
assembling the tracked observations comprising the common agent ID into the observations of the time-sequence of positions.

### Claim 13 (depends on 11)

The computer-readable medium of claim 11, wherein tracking the observations of the agent identified across the series of image frames within the image data comprises:
based on (i) one or more respective prior positions associated with the one or more previous observations of the agent and (ii) a motion model for the agent, determining a predicted position of the agent; and
determining a match between the predicted position of the agent and a future observation of the agent based on:
an intersection of the predicted position of the agent and the future observation of the agent; and
a visual matching analysis of the predicted position of the agent with the future observation of the agent.

### Claim 14 (depends on 11)

The computer-readable medium of claim 11, wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:
determine a respective measurement error associated with each observation in the observations of the time-sequence of positions that is based on a measured distance of the positions from the vehicle; and
incorporate the respective error value associated with each observation into at least the first optimization operation or the second optimization operation.

### Claim 15 (depends on 14)

The computer-readable medium of claim 14, wherein the respective error value increases quadratically as the measured distance from the vehicle increases.

### Claim 16 (depends on 11)

The computer-readable medium of claim 11, wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:
augment the second updated time-sequence of observations to include one or more additional intermediate observations between each pair of observations in the second updated time-sequence of observations.

### Claim 17 (depends on 16)

The computer-readable medium of claim 16, wherein the image data is captured by a stereo camera having a frame rate of 5 frames per second, and wherein augmenting the second updated time-sequence of observations comprises increasing the frame rate of the second updated time-sequence of observations from 5 frames per second to 30 frames per second.

### Claim 18 (depends on 11)

The computer-readable medium of claim 11, wherein beginning with the position associated with the observation having the highest confidence in the time-sequence of observations and proceeding in the first direction comprises:
determining that one of (i) an earliest position in the time-sequence of positions or (ii) a latest position in the time-sequence of position is the position of the agent associated with the observation having the highest confidence in the time-sequence of observations based on the measured distance of each respective position from the vehicle; and
proceeding (i) chronologically forward or (ii) chronologically backward, respectively.

### Claim 19 (depends on 11)

The computer-readable medium of claim 11, wherein generating the first updated time-sequence of positions of the agent comprises processing at least two or more positions of the observed time-sequence of positions by beginning with a position associated with an earliest observation in the time-sequence of observations and proceeding chronologically forward; and
wherein generating the second updated time-sequence of positions of the agent comprises processing at least two or more positions of the first updated time-sequence of observations by beginning with a position associated with a latest observation and proceeding chronologically backward.

### Claim 20

A computing system comprising:
at least one processor;
a non-transitory computer-readable medium; and
program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:
based on image data captured by a vehicle in an environment, obtaining observations of a time-sequence of positions of an agent identified within the image data;
generating a first updated time-sequence of positions of the agent by performing a first optimization operation on the observed time-sequence of positions, wherein the first optimization operation includes processing at least two or more positions of the observed time-sequence of positions by beginning with a position associated with an observation having the highest confidence in the time-sequence of observations and proceeding in a first direction;
after generating the first updated time-sequence of positions, generating a second updated time-sequence of positions of the agent by performing a second optimization operation on the first updated time-sequence of positions, wherein the second optimization operation includes processing at least two or more positions of the first updated time-sequence of positions in a second direction opposite the first direction; and
deriving a trajectory for the agent in the environment based on the second updated time-sequence of positions for the agent.
