# US 2021/0343034 A1 · Systems and methods for maneuvering a vehicle responsive to detecting a condition based on dynamic object trajectories

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343034 A1
- Document ID: US20210343034A1
- Status: Published application
- Kind: A1
- Application number: 17/322,574
- Filed: 2021-05-17
- Published: 2021-11-04
- Earliest claimed: 2015-12-18
- Assignees: Iris Automation, Inc.
- Applicants: Iris Automation, Inc.
- Inventors: Alexander Harmsen; James Howard; Patricio Alejandro Galindo
- Web page: https://patentgrep.com/patent/US20210343034A1

## CPC

- G06T 7/579: from motion
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G05D 1/106: {Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones}
- G05D 1/606: Compensating for or utilising external environmental conditions, e.g. wind or water currents (station keeping G05D1/611)
- G05D 1/689: Pointing payloads towards fixed or moving targets (positioning towed, pushed or suspended implements G05D1/672)
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/292: Multi-camera tracking
- G06T 17/05: Geographic models
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04N 7/183: {for receiving images from a single remote source}
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G08G 1/16: Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08)
- G08G 3/02: Anti-collision systems
- G08G 5/80: Anti-collision systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

A self-contained, low-cost, low-weight guidance system for vehicles is provided. The guidance system can include an optical camera, a case, a processor, a connection between the processor and an on-board control system, and computer algorithms running on the processor. The guidance system can be integrated with a vehicle control system through “plug and play” functionality or a more open Software Development Kit. The computer algorithms re-create 3D structures as the vehicle travels and continuously updates a 3D model of the environment. The guidance system continuously identifies and tracks terrain, static objects, and dynamic objects through real-time camera images. The guidance system can receive inputs from the camera and the onboard control system. The guidance system can be used to assist vehicle navigation and to avoid possible collisions. The guidance system can communicate with the control system and provide navigational direction to the control system.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method, comprising:
receiving, by a computing device including one or more processors, a sequence of images captured by an image capture device mounted on a movable entity;
identifying, by the computing device, control data sent to a vehicle drive system of the movable entity from a control system of the movable entity;
determining, by the computing device, motion data comprising velocity values or acceleration values of the movable entity that correspond to when each of the sequence of images were captured by the image capture device;
mapping, by the computing device, the motion data of the movable entity to the control data;
tracking, by the computing device, a plurality of points of interest corresponding to an object across the sequence of images;
identifying, by the computing device, based on tracking the plurality of points of interest, a dynamic object represented in the sequence of images by comparing actual positions of each point of interest in the sequence of images relative to expected positions of each point of interest calculated based on the velocity values or the acceleration values corresponding to each of the sequence of images;
determining, by the computing device, a predicted trajectory of the dynamic object based on movement of the plurality of points of interest across the sequence of images;
detecting, by the computing device, a maneuver condition based on the mapping of the control data to the motion data and the predicted trajectory of the dynamic object;
generating, by the computing device, control instructions that, when executed by the control system of the movable entity, cause the movable entity to maneuver according to the maneuver condition; and
transmitting, by the computing device to the control system of the movable entity, the control instructions, causing the movable entity to maneuver according to the maneuver condition.

### Claim 22 (depends on 21)

The method of claim 21, wherein determining the motion data of the movable entity includes receiving the motion data from the movable entity via a communication link.

### Claim 23 (depends on 21)

The method of claim 21, wherein determining the motion data of the movable entity further comprises determining the velocity values using a triangulation technique.

### Claim 24 (depends on 21)

The method of claim 21, wherein tracking the plurality of points of interest corresponding to the object further comprises tracking, by the computing device, movement of a plurality of pixels of the sequence of images using a respective parameter value of a respective pixel of the plurality of pixels.

### Claim 25 (depends on 21)

The method of claim 21, wherein identifying the dynamic object in the sequence of images further comprises determining, by the computing device, the expected positions of each point of interest further based on an expected motion profile of the point of interest.

### Claim 26 (depends on 21)

The method of claim 21, wherein tracking the plurality of points of interest further comprises identifying, by the computing device, the plurality of points of interest by applying a contrast filter to identify a respective pixel in the sequence of images that has a contrast ratio relative to pixels within a predetermined distance of the respective pixel that exceeds a threshold.

### Claim 27 (depends on 21)

The method of claim 21, wherein identifying the dynamic object further comprises determining, by the computing device, for each point of interest of the plurality of points of interest, a distance the point of interest moved across the sequence of images.

### Claim 28 (depends on 21)

The method of claim 21, further comprising determining, by the computing device, a trajectory of the movable entity, and wherein detecting the maneuver condition is further based on the trajectory of the movable entity.

### Claim 29 (depends on 21)

The method of claim 21, further comprising:
estimating, by the computing device, responsive to tracking the plurality of points of interest across the sequence of images and based on the motion data of the movable entity, a depth value for each point of interest of the plurality of points of interest; and
generating, by the computing device, a 3D world map based on the depth value of each point of interest of the plurality of points of interest.

### Claim 30 (depends on 29)

The method of claim 29, further comprising generating, by the computing device, a visual situational awareness model based on the 3D world map and the dynamic object, and wherein detecting the maneuver condition is based on detecting a change in the visual situational awareness model.

### Claim 31

A system, comprising:
one or more processors coupled to memory, the one or more processors configured to:
receive a sequence of images captured by an image capture device mounted on a movable entity;
identify control data sent to a vehicle drive system of the movable entity from a control system of the movable entity;
determine motion data comprising velocity values or acceleration values of the movable entity that correspond to when each of the sequence of images were captured by the image capture device;
map the motion data of the movable entity to the control data;
track a plurality of points of interest corresponding to an object across the sequence of images;
identify, based on tracking the plurality of points of interest, a dynamic object represented in the sequence of images by comparing actual positions of each point of interest in the sequence of images relative to expected positions of each point of interest calculated based on the velocity values or the acceleration values corresponding to each of the sequence of images;
determine, a predicted trajectory of the dynamic object based on movement of the plurality of points of interest across the sequence of images;
detect a maneuver condition based on the mapping of the control data to the motion data and the predicted trajectory of the dynamic object;
generate control instructions that, when executed by the control system of the movable entity, cause the movable entity to maneuver according to the maneuver condition; and
transmit, to the control system of the movable entity, the control instructions, causing the movable entity to maneuver according to the maneuver condition.

### Claim 32 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to determine the motion data of the movable entity by performing operations comprising receiving the motion data from the movable entity via a communication link.

### Claim 33 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to determine the motion data of the movable entity by performing operations comprising determining the velocity values using a triangulation technique.

### Claim 34 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to track the plurality of points of interest corresponding to the object by performing operations comprising tracking movement of a plurality of pixels of the sequence of images using a respective parameter value of a respective pixel of the plurality of pixels.

### Claim 35 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to identify the dynamic object in the sequence of images by performing operations comprising determining the expected positions of each point of interest further based on an expected motion profile of the point of interest.

### Claim 36 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to track the plurality of points of interest by performing operations comprising identifying the plurality of points of interest by applying a contrast filter to identify a respective pixel in the sequence of images that has a contrast ratio relative to pixels within a predetermined distance of the respective pixel that exceeds a threshold.

### Claim 37 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to identify the dynamic object by performing operations comprising determining, for each point of interest of the plurality of points of interest, a distance the point of interest moved across the sequence of images.

### Claim 38 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to determine a trajectory of the movable entity, and detect the maneuver condition further based on the trajectory of the movable entity.

### Claim 39 (depends on 31)

The system of claim 31, wherein the one or more processors are further configured to:
estimate, responsive to tracking the plurality of points of interest across the sequence of images and based on the motion data of the movable entity, a depth value for each point of interest of the plurality of points of interest; and
generate a 3D world map based on the depth value of each point of interest of the plurality of points of interest.

### Claim 40 (depends on 39)

The system of claim 39, wherein the one or more processors are further configured to generate a visual situational awareness model based on the 3D world map and the dynamic object, and wherein detecting the maneuver condition is based on detecting a change in the visual situational awareness model.
