# US 10,769,957 B2 · Systems and methods for vehicle guidance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,957 B2
- Document ID: US10769957B2
- Status: Granted
- Kind: B2
- Application number: 16/360,097
- Filed: 2019-03-21
- Published: 2019-07-11
- Granted: 2020-09-08
- Earliest claimed: 2015-08-11
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Pascal Gohl; Sammy Omari
- Web page: https://patentgrep.com/patent/US10769957B2

## CPC

- G08G 5/80: Anti-collision systems
- B64C 39/024: {of the remote controlled vehicle type, i.e. RPV}
- B64D 47/08: Arrangements of cameras
- B64U 10/13: Flying platforms
- G05D 1/106: {Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/593: from stereo images
- G06T 7/60: Analysis of geometric attributes
- G06T 7/73: using feature-based methods
- G06V 20/13: Satellite images
- G06V 20/17: taken from planes or by drones
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 5/55: for a single aircraft
- G08G 5/57: for unmanned aircraft
- H04N 13/128: Adjusting depth or disparity
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- G06T 2200/04: involving 3D image data
- G06T 2207/10012: Stereo images
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/30168: Image quality inspection
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B64C: AEROPLANES; HELICOPTERS
- B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

This disclosure relates to systems and methods for vehicle guidance. Stereo images may be obtained at different times using a stereo image sensor. A depth image may be determined based on an earlier obtained pair of stereo images. The depth image may be refined based on predictions of an earlier stereo image and a later obtained stereo image. Depth information for an environment around a vehicle may be obtained. The depth information may characterize distances between the vehicle and the environment around the vehicle. A spherical depth map may be generated from the depth information. Maneuver controls for the vehicle may be provided based on the spherical depth map.

## Claims

### Claim 1

An unmanned aerial vehicle, the unmanned aerial vehicle comprising:
a sensor generating visual information from an environment around the unmanned aerial vehicle; and
a processing apparatus coupled to a memory storing instructions that when executed causes the processing apparatus to:
generate a spherical depth map from depth information associated with the environment, the spherical depth map representing distances to closest surfaces of the environment around the unmanned aerial vehicle as a function of longitude angles and latitude angles, the spherical depth map being divided into map cells corresponding to the longitude angles and the latitude angles, the map cells including a first map cell corresponding to a first longitude angle and a first latitude angle; and
provide maneuver controls for the unmanned aerial vehicle based on the spherical depth map.

### Claim 2 (depends on 1)

The unmanned aerial vehicle of claim 1, wherein the sensor is a stereo image sensor comprising:
a first image sensor configured to generate first visual output signals conveying first visual information within a first field of view of the first image sensor; and
a second image sensor configured to generate second visual output signals conveying second visual information within a second field of view of the second image sensor, wherein the depth information is determined by comparing the first visual information with the second visual information.

### Claim 3 (depends on 1)

The unmanned aerial vehicle of claim 1, wherein generating the spherical depth map includes determining distance values for the map cells, the distance values determined based on the distances to the closest surfaces of the environment around the unmanned aerial vehicle, wherein a first distance value for the first map cell is determined based on a first distance to a first closest surface of the environment around the unmanned aerial vehicle at the first longitude angle and the first latitude angle.

### Claim 4 (depends on 3)

The unmanned aerial vehicle of claim 3, wherein the determination of the distance values disregards distances to surfaces of the environment around the unmanned aerial vehicle at individual longitude angles and individual latitude angles that are greater than the distances to the closest surfaces of the environment around the unmanned aerial vehicle at the individual longitude angles and the individual latitude angles.

### Claim 5 (depends on 3)

The unmanned aerial vehicle of claim 3, wherein the distance values correspond only to the closest surfaces of the environment around the unmanned aerial vehicle at individual longitude angles and individual latitude angles.

### Claim 6 (depends on 3)

The unmanned aerial vehicle of claim 3, wherein the spherical depth map includes a dead zone in a polar region, and the distance values are not determined for the map cells in the dead zone.

### Claim 7 (depends on 1)

The unmanned aerial vehicle of claim 1, wherein the processing apparatus comprises a hardware-implemented processor and a software-implemented processor.

### Claim 8 (depends on 7)

The unmanned aerial vehicle of claim 7, wherein the hardware-implemented processor is located remotely from the software-implemented processor.

### Claim 9 (depends on 1)

The unmanned aerial vehicle of claim 1, further including instructions that when executed causes the processing apparatus to:
responsive to detecting the unmanned aerial vehicle traveling a threshold distance, transform the spherical depth map such that the center of the spherical depth map coincides with a location of the unmanned aerial vehicle.

### Claim 10

A method for collision avoidance of an unmanned aerial vehicle, the method comprising:
generating visual information from an environment around the unmanned aerial vehicle;
generating a spherical depth map from depth information associated with the environment, the spherical depth map representing distances to closest surfaces of the environment around the unmanned aerial vehicle as a function of longitude angles and latitude angles, the spherical depth map being divided into map cells corresponding to the longitude angles and the latitude angles, the map cells including a first map cell corresponding to a first longitude angle and a first latitude angle; and
providing maneuver controls for the unmanned aerial vehicle based on the spherical depth map.

### Claim 11 (depends on 10)

The method of claim 10, wherein a stereo image sensor of the unmanned aerial vehicle generates the visual information from the environment around the unmanned aerial vehicle, further wherein the stereo image sensor comprises:
a first image sensor configured to generate first visual output signals conveying first visual information within a first field of view of the first image sensor; and
a second image sensor configured to generate second visual output signals conveying second visual information within a second field of view of the second image sensor.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
determining the depth information by comparing the first visual information with the second visual information.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
generating the spherical depth map by determining distance values for the map cells, the distance values determined based on the distances to the closest surfaces of the environment around the unmanned aerial vehicle, wherein a first distance value for the first map cell is determined based on a first distance to a first closest surface of the environment around the unmanned aerial vehicle at the first longitude angle and the first latitude angle.

### Claim 14 (depends on 13)

The method of claim 13, wherein the determination of the distance values disregards distances to surfaces of the environment around the unmanned aerial vehicle at individual longitude angles and individual latitude angles that are greater than the distances to the closest surfaces of the environment around the unmanned aerial vehicle at the individual longitude angles and the individual latitude angles.

### Claim 15 (depends on 13)

The method of claim 13, wherein the distance values correspond only to the closest surfaces of the environment around the unmanned aerial vehicle at individual longitude angles and individual latitude angles.

### Claim 16 (depends on 13)

The method of claim 13, wherein the spherical depth map includes a dead zone in a polar region, and the distance values are not determined for the map cells in the dead zone.

### Claim 17 (depends on 10)

The method of claim 10, wherein the unmanned aerial vehicle includes a processing apparatus that comprises a hardware-implemented processor and a software-implemented processor.

### Claim 18 (depends on 17)

The method of claim 17, wherein the hardware-implemented processor is located remotely from the software-implemented processor.

### Claim 19 (depends on 10)

The method of claim 10, further comprising:
responsive to detecting the unmanned aerial vehicle traveling a threshold distance, transform the spherical depth map such that the center of the spherical depth map coincides with a location of the unmanned aerial vehicle.

### Claim 20 (depends on 10)

The method of claim 10, wherein dividing the spherical depth map into map cells enables mapping of the depth information to a fixed sized grid of discrete bearings.