# US 12,001,207 B2 · System for controlling an autonomous driving vehicle or air vessel, which can be controlled on the basis of steering and acceleration values, and an autonomous driving vehicle or air vessel provided with such a system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,207 B2
- Document ID: US12001207B2
- Status: Granted
- Kind: B2
- Application number: 16/987,612
- Filed: 2020-08-07
- Published: 2021-02-11
- Granted: 2024-06-04
- Earliest claimed: 2019-08-09
- Applicants: Stephan Johannes Smit; Johannes Wilhelmus Maria Van Bentum
- Inventors: Stephan Johannes Smit; Johannes Wilhelmus Maria Van Bentum
- Web page: https://patentgrep.com/patent/US12001207B2

## CPC

- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- B60W 30/06: Automatic manoeuvring for parking
- B60W 60/0015: {specially adapted for safety}
- G01C 21/3407: {specially adapted for specific applications}
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G05D 1/101: {specially adapted for aircraft}
- G05D 1/2435: {Extracting 3D information}
- G05D 1/246: using environment maps, e.g. simultaneous localisation and mapping [SLAM]
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/46: Control of position or course in three dimensions [3D]
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G08G 5/32: for flight plan preparation
- G08G 5/50: Navigation or guidance aids
- G08G 5/55: for a single aircraft
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2520/00: Input parameters relating to overall vehicle dynamics
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

System for controlling an autonomous vehicle on the basis of control values and acceleration values, having a safety-determining module configured to receive live images from a camera, to receive recorded stored images preprocessed for image recognition from an internal safety-determining module data storage, to receive navigation instruction(s) from a navigation module; to compare the live images with the stored images to determine a degree of correspondence; and to determine a safety value which indicates the extent to which the determined degree of correspondence suffices to execute the navigation instruction(s); wherein a control module is configured to receive the navigation instruction(s); to receive the live images; to receive the safety value; to compare the safety value to a predetermined value; and if the safety value is greater than the predetermined value, to convert the navigation instruction(s) and the camera images into control values and acceleration values for the autonomous vehicle.

## Claims

### Claim 1

A system for controlling a vehicle, wherein the vehicle comprises a land-based vehicle or an aircraft, the vehicle which is configured to be controlled to move autonomously based on control values and acceleration values, the system comprising:
a navigation module;
a control module;
at least one camera;
a recognition module;
a safety-determining module;
a communication module;
a data storage for camera images, wherein the navigation module, the control module, the at least one camera, the recognition module, the safety-determining module, the communication module, and the data storage are communicatively coupled, the system configured to execute a method comprising:
obtaining, by the navigation module, a destination for the vehicle, wherein the destination is selected from a finite list of destinations;
determining, by the navigation module, a current location of the vehicle based on obtaining a last known location of the vehicle;
generating, by the navigation module, a route from the current location of the vehicle to the destination, wherein the route comprises one or more navigational points;
converting, by the navigation module, the route into one or more navigational instructions, wherein each navigational instruction of the navigational instructions comprises a terminating navigational point selected from the navigational points;
continuously obtaining, by the at least one camera, images, wherein the at least one camera captures images from the vehicle;
transmitting, by the at least one camera, the images to the navigation module, the control module, the recognition module, and the safety-determining module;
storing, by the safety-determining module, the images in the data storage, wherein the storing comprises pre-processing the images for image recognition;
based on the storing, annotating, by the safety-determining module, each stored image of the stored images in the data storage with an annotation representing the navigation point of the navigational points representing a physical position of the vehicle when each image was obtained;
obtaining, by the control module, a control value and an acceleration value for at least one navigational instruction of the navigational instructions, wherein the control value and the acceleration value comprise movement instructions to control the vehicle to execute the at least one navigational instruction, the obtaining comprising:
obtaining, by the safety-determining module, a live image from the camera;
comparing, by the safety-determining module, the live image to a portion of the stored images;
based on the comparing, by the safety-determining module, determining by the control module, a degree of correspondence between the live image and the portion of the stored images;
based on the degree of correspondence, generating, by the safety-determining module, a safety value;
comparing, by the control module, the safety value to a predetermined value;
based on determining that the safety value is greater than the predetermined value, obtaining the control value and the acceleration value, the obtaining comprising:
converting, by the control module, the at least one navigational instruction to the control value and the acceleration value; and
based on determining that the safety value is less than or equal to the predetermined value, requesting, by the control module, the control value and the acceleration value and obtaining the control value and the acceleration value responsive to the requesting;
updating, by the control module, at least one stored image of the stored images in the data storage with the acceleration value;
determining, by the control module, whether to execute the at least one navigational instruction, the determining comprising:
controlling, by the control module, the vehicle to execute the at least one navigational instruction, wherein the executing is based on the control value and the acceleration value;
obtaining, by the recognition module, a current image from the camera after the controlling;
confirming, by the recognition module, the executing of the at least one navigational instruction based on analyzing the current image to determine that the vehicle has reached the terminating navigational point of the at least one navigational instruction; and
transmitting, by the recognition module, the confirming to the navigation module.

### Claim 2 (depends on 1)

The system according to claim 1, wherein based on determining that the safety value is less than or equal to the predetermined value, stopping or parking, by the control module, the vehicle while waiting for the control value or the acceleration value.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the requesting is via the communication module.

### Claim 4 (depends on 1)

The system according to claim 1, wherein comparing the live image to the portion of the stored images comprises:
determining recognition points in the live image and in the stored images; and
comparing the recognition points in the live image with recognition points determined in the stored images.

### Claim 5 (depends on 1)

The system according to claim 1, wherein the control value comprises a directional value, the converting further comprising:
accessing, by the control module, a Nvidia Dave 2 network topology; and
utilizing, by the control module, the for the Nvidia Dave 2 network topology to perform deep learning to covert the at least one navigational instruction into the directional value and the acceleration value.

### Claim 6 (depends on 1)

The system according to claim 1, wherein controlling the vehicle further comprises:
providing, by the control module, the control value at a frequency of at least 2 Hz, and the acceleration value at a frequency of at least 10 Hz.

### Claim 7 (depends on 1)

The system according to claim 1, further comprising a clear-passage-determining module communicatively coupled to the at least one camera and the navigational module, the method further comprising:
annotating, by the clear-passage-determining module, each stored image of the stored images in the data storage with a value which indicates a chance of clear passage.

### Claim 8 (depends on 7)

A system according to claim 7, wherein the determining whether to execute the at least one navigational instruction, further comprises, prior to the controlling the vehicle to execute the at least one navigational instruction:
determining, by the clear-passage-determining module, if the at least one navigation instruction can be executed, the determining comprising:
receiving, by the clear-passage-determining module, the stored images and annotations, wherein the annotations for each image comprise the value which indicates the chance of clear passage from the data storage;
comparing, by the clear-passage-determining module, the live image with the stored images; and
determining, by the clear-passage-determining module, based on the at least one navigation instruction and the annotations, if the navigation instruction can be executed.

### Claim 9 (depends on 8)

The system according to claim 8, the method further comprising:
based on determining that the navigation instruction cannot be executed, obtaining, by the clear-passage-determining module, a new live image from the at least one camera;
comparing, by the clear-passage-determining module, the new live image with the stored images; and
determining, by the clear-passage-determining module, whether the navigation instruction can be executed based on the at least one navigation instructions and the annotations.

### Claim 10 (depends on 7)

The system according to claim 7, wherein determining if the navigation instruction can be executed comprises determining, based on the at least one navigation instructions and the annotations, a clear-passage-safety value which indicates a degree of certainty that the at least one navigation instruction can be executed.

### Claim 11 (depends on 8)

The system according to claim 8, wherein determining, by the clear-passage-determining module, if the navigation instruction can be executed comprises determining, by the clear-passage-determining module, based on the at least one navigation instruction and the annotations, a clear-passage-safety value which indicates a degree of certainty that the at least one navigation instruction can be executed.

### Claim 12 (depends on 9)

The system according to claim 9, wherein determining, by the clear-passage-determining module, if the navigation instruction can be executed comprises determining, by the clear-passage-determining module, based on the at least one navigation instruction and the annotations, a clear-passage-safety value which indicates a degree of certainty that the at least one navigation instruction can be executed.

### Claim 13

A computer-implemented method to control a vehicle, wherein the vehicle comprises a land-based vehicle or an aircraft, the method comprising:
obtaining, by one or more processors, a destination for the vehicle, wherein the destination is selected from a finite list of destinations;
determining, by the one or more processors, a current location of the vehicle based on obtaining a last known location of the vehicle;
generating, by the one or more processors, a route from the current location of the vehicle to the destination, wherein the route comprises one or more navigational points;
converting, by the one or more processors, the route into one or more navigational instructions, wherein each navigational instruction of the navigational instructions comprises a terminating navigational point selected from the navigational points;
continuously obtaining, by the one or more processors, images from a camera, wherein the at least one camera captures images from the vehicle;
storing, by the one or more processors, the images in a data storage;
based on the storing, annotating, by the one or more processors, each stored image of the stored images in the data storage with an annotation representing the navigation point of the navigational points representing a physical position of the vehicle when each image was obtained;
obtaining, by the one or more processors, a control value and an acceleration value for at least one navigational instruction of the navigational instructions, wherein the control value and the acceleration value comprise movement instructions to control the vehicle to execute the at least one navigational instruction, the obtaining comprising:
obtaining, by the one or more processors, a live image from the camera;
comparing, by the one or more processors, the live image to a portion of the stored images;
based on the comparing, determining, by the one or more processors, a degree of correspondence between the live image and the portion;
based on the degree of correspondence, generating, by the one or more processors, a safety value;
comparing, by the one or more processors, the safety value to a predetermined value;
based on determining that the safety value is greater than the predetermined value, obtaining the control value and the acceleration value based on converting the at least one navigational instruction to the control value and the acceleration value; and
based on determining that the safety value is less than or equal to the predetermined value, requesting, by the one or more processors, the control value and the acceleration value and obtaining the control value and the acceleration value responsive to the requesting;
updating, by the one or more processors, at least one stored image of the stored images in the data storage with the acceleration value;
controlling, by the one or more processors, the vehicle to execute the at least one navigational instruction, wherein the executing is based on the control value and the acceleration value;
obtaining, by the one or more processors, a current image from the camera after the controlling; and
confirming, by the one or more processors, the executing of the at least one navigational instruction based on analyzing the current image to determine that the vehicle has reached the terminating navigational point of the at least one navigational instruction.