# US 11,300,965 B2 · Methods and systems for navigating autonomous and semi-autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,300,965 B2
- Document ID: US11300965B2
- Status: Granted
- Kind: B2
- Application number: 16/584,566
- Filed: 2019-09-26
- Published: 2020-04-02
- Granted: 2022-04-12
- Earliest claimed: 2018-10-01
- Assignees: EmergeX, LLC
- Applicants: EmergeX, LLC
- Inventors: Roy Feinson; Ari Mikhael Katz
- Web page: https://patentgrep.com/patent/US11300965B2

## CPC

- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- G05D 1/0221: {involving a learning process}
- G05D 1/0238: {using obstacle or wall sensors (G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93)}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/2435: {Extracting 3D information}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80)
- G05D 1/628: following the obstacle profile, e.g. a wall or undulated terrain
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/02: Neural networks
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06V 2201/02: Recognising information on displays, dials, clocks
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems are disclosed for an improved control system in autonomous and semi-autonomous vehicles. More specifically, the methods and systems relate to powering control systems of autonomous and semi-autonomous vehicles through the use of computer vision based on delta images (i.e., delta-vision).

## Claims

### Claim 1

A method of identifying objects while operating autonomous or semi-autonomous vehicles, the method comprising:
receiving, using control circuitry, a first delta image of a first vehicular scenario;
generating, using the control circuitry, a first pixel array based on the image of the first vehicular scenario;
labeling, using the control circuitry, the first pixel array with a known object in the first vehicular scenario;
training, using the control circuitry, an artificial neural network to detect the known object in the first vehicular scenario based on the labeled first pixel array;
receiving, using the control circuitry, a second delta image of a second vehicular scenario;
generating, using the control circuitry, a second pixel array based on the second delta image of the second vehicular scenario;
inputting, using the control circuitry, the second pixel array into the trained neural network; and
receiving, using the control circuitry, an output from the trained neural network indicating that the second vehicular scenario includes the known object.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
storing, using a camera of an autonomous or semi-autonomous vehicle, an image of the second vehicular scenario;
rotating the camera;
storing, using the camera of an autonomous or semi-autonomous vehicle, a rotated image of the second vehicular scenario; and
generating the second delta image based on the image and the rotated image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving an image of the second vehicular scenario and delta image criterion;
selecting a look back image based on the criterion, wherein the look back image is a previously stored image of the first vehicular scenario;
retrieving the look back image; and
generating the second delta image based on the look back image.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
generating a periphery image based on the second delta image by removing chromatic information from the second delta image;
determining a pixel density of a portion of the periphery image;
comparing the pixel density to a threshold density; and
in response to the pixel density equaling or exceeding the threshold, selecting a portion of the second delta image corresponding to the portion for generating the second pixel array.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining a dangerous condition based on the output;
determining an area in the second delta image with no delta information; and
directing an autonomous or semi-autonomous vehicle to the area based the second delta image having no delta information in that area.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining a thickness of a delta edge of the known object in the second delta image; and
determining a speed of the known object or a distance to the known object based on the thickness.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
identifying the known object based on the output;
retrieving a size of the known object;
mapping a terrain of the second vehicular scenario based on the size and distance of the known object.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining a first thickness of a delta edge of the known object in the second delta image;
receiving a third delta image of the second vehicular scenario;
determining a second thickness of a delta edge of the known object in a third delta image;
comparing the first thickness and the second thickness; and
determining an acceleration or deceleration of the known object based on the comparison.

### Claim 9

A system for identifying objects while operating autonomous or semi-autonomous vehicles, the system comprising:
memory configured to store an artificial neural network; and
control circuitry configured to:
receive a first delta image of a first vehicular scenario;
generate a first pixel array based on the image of the first vehicular scenario;
label the first pixel array with a known object in the first vehicular scenario;
train artificial neural network to detect the known object in the first vehicular scenario based on the labeled first pixel array;
receive a second delta image of a second vehicular scenario;
generate a second pixel array based on the second delta image of the second vehicular scenario;
input the second pixel array into the trained neural network; and
receive an output from the trained neural network indicating that the second vehicular scenario includes the known object.

### Claim 10 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
store, using a camera of an autonomous or semi-autonomous vehicle, an image of the second vehicular scenario;
rotate the camera;
store, using the camera of an autonomous or semi-autonomous vehicle, a rotated image of the second vehicular scenario;
generate the second delta image based on the image and the rotated image.

### Claim 11 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
receive an image of the second vehicular scenario and delta image criterion;
select a look back image based on the criterion, wherein the look back image is a previously stored image of the first vehicular scenario
retrieve the look back image; and
generate the second delta image based on the look back image.

### Claim 12 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
generate a periphery image based on the second delta image by removing chromatic information from the second delta image;
determine a pixel density of a portion of the periphery image;
compare the pixel density to a threshold density; and
select a portion of the second delta image corresponding to the portion for generating the second pixel array in response to the pixel density equaling or exceeding the threshold.

### Claim 13 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
determine a dangerous condition based on the output;
determine an area in the second delta image with no delta information; and
direct an autonomous or semi-autonomous vehicle to the area based the second delta image having no delta information in that area.

### Claim 14 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
determine a thickness of a delta edge of the known object in the second delta image; and
determine a speed of the known object or a distance to the known object based on the thickness.

### Claim 15 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
identify the known object based on the output;
retrieve a size of the known object;
map a terrain of the second vehicular scenario based on the size and distance of the known object.

### Claim 16 (depends on 9)

The system of claim 9, wherein the control circuitry is further configured to:
determine a first thickness of a delta edge of the known object in the second delta image;
receive a third delta image of the second vehicular scenario;
determine a second thickness of a delta edge of the known object in a third delta image;
compare the first thickness and the second thickness; and
determine an acceleration or deceleration of the known object based on the comparison.

### Claim 17

A method of identifying objects while operating autonomous or semi-autonomous vehicles, the method comprising:
receiving, using control circuitry of an autonomous or semi-autonomous vehicle, a delta image of a vehicular scenario;
generating, using the control circuitry, a pixel array based on the delta image;
inputting, using the control circuitry, the pixel array into a neural network;
receiving, using the control circuitry, an output from the neural network identifying an object in the pixel array; and
initiating a control operation for the autonomous or semi-autonomous vehicle based on the object.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
storing an image of the object;
rotating the image of the object;
storing a rotated image of the object;
generating a test delta image based on the image and the rotated image;
labeling the test delta image with the object; and
training the neural network based on the labeled test delta image.

### Claim 19 (depends on 17)

The method of claim 17, further comprising:
generating a periphery image based on the delta image by removing chromatic information from the delta image;
determining a pixel density of a portion of the periphery image;
comparing the pixel density to a threshold density; and
in response to the pixel density equaling or exceeding the threshold, inputting the portion into the neural network.

### Claim 20 (depends on 17)

The method of claim 17, further comprising:
receiving an image of the vehicular scenario and delta image criterion;
selecting a look back image based on the criterion, wherein the look back image is a previously stored image of the vehicular scenario;
retrieving the look back image; and
generating the delta image based on the look back image.

### Claim 21 (depends on 17)

The method of claim 17, wherein the control operation comprises:
determining a thickness of a delta edge of the object in the delta image; and
determining a distance to the object based on the thickness.