# US 2021/0241420 A1 · Method, apparatus, and computer program product for generating an overhead view of an environment from a perspective image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241420 A1
- Document ID: US20210241420A1
- Status: Published application
- Kind: A1
- Application number: 17/302,114
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2018-12-21
- Assignees: HERE Global B.V.
- Applicants: Here Global B.V.
- Inventors: Anirudh VISWANATHAN
- Web page: https://patentgrep.com/patent/US20210241420A1

## CPC

- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G01C 21/3667: {Display of a road map (G01C21/3614 takes precedence; guidance using 3D or perspective road maps G01C21/3635)}
- G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles}
- G01C 21/3852: {Data derived from aerial or satellite images}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20212: Image combination
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

A method, apparatus and computer program product are provided for warping a perspective image into the ground plane using a homography transformation to estimate a bird's eye view in real time. Methods may include: receiving first sensor data from a first vehicle traveling along a road segment in an environment, where the first sensor data includes perspective image data of the environment, and where the first sensor data includes a location and a heading; retrieving a satellite image associated with the location and heading; applying a deep neural network to regress a bird's eye view image from the perspective image data; applying a Generative Adversarial Network (GAN) to the regressed bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image; and deriving values of a homography matrix between the sensor data and the established bird's eye view image.

## Claims

### Claim 1

An apparatus comprising at least one processor and at least one non-transitory memory including computer program code instructions, the computer program code instructions configured to, when executed, cause the apparatus to at least:
receive first sensor data from a first sensor in an environment, wherein the first sensor data comprises perspective image data of the environment, wherein the first sensor data comprises a location and a heading;
retrieve a satellite image of the environment associated with the location and heading, wherein the satellite image is captured from a position above a ground plane; and
generate a bird's eye view image of the environment above the ground plane from the first sensor data by warping the first sensor data from the perspective image data of the environment based on processing of the first sensor data relative to the satellite image.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein processing of the first sensor data relative to the satellite image comprises causing the apparatus to input the first sensor data and the satellite image into a deep neural network to regress the bird's eye view image.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the apparatus is further caused to apply a Generative Adversarial Network (GAN) to the bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the apparatus is further caused to:
derive values of a homography matrix between the first sensor data and the stabilized bird's eye view image; and
store the derived values of the homography matrix.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the apparatus is further caused to:
receive second sensor data from a second sensor in a second environment, wherein the second sensor data comprises perspective image data of the second environment from a perspective of the second sensor;
apply a homography matrix including the stored, derived values to the second sensor data to warp the perspective image data of the second environment into a second ground plane;
generate a second bird's eye view of the second environment from the perspective image data of the second environment; and
provide for presentation of the second bird's eye view of the second environment on a display.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the apparatus is further caused to augment satellite image data of the second environment with the second bird's eye view of the second environment.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the bird's eye view image of the environment includes dynamic objects not found in a satellite image of the environment.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein causing the apparatus to retrieve a satellite image of the environment associated with the location and heading comprises causing the apparatus to identify the location in a satellite image map database and to identify a portion of the satellite image corresponding to the heading.

### Claim 9

A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code portions stored therein, the computer-executable program code portions comprising program code instructions configured to:
receive first sensor data from a first sensor in an environment, wherein the first sensor data comprises perspective image data of the environment, wherein the first sensor data comprises a location and a heading;
retrieve a satellite image of the environment associated with the location and heading, wherein the satellite image is captured from a position above a ground plane; and
generate a bird's eye view image of the environment above the ground plane from the first sensor data by warping the first sensor data from the perspective image data of the environment based on processing of the first sensor data relative to the satellite image.

### Claim 10 (depends on 9)

The computer program product of claim 9, wherein processing of the first sensor data relative to the satellite image comprise program code instructions to input the first sensor data and the satellite image into a deep neural network to regress the bird's eye view image.

### Claim 11 (depends on 9)

The computer program product of claim 9, further comprising program code instructions to apply a Generative Adversarial Network (GAN) to the bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image.

### Claim 12 (depends on 11)

The computer program product of claim 11, further comprising program code instructions to:
derive values of a homography matrix between the first sensor data and the stabilized bird's eye view image; and
store the derived values of the homography matrix.

### Claim 13 (depends on 12)

The computer program product of claim 12, further comprising program code instructions to:
receive second sensor data from a second sensor in a second environment, wherein the second sensor data comprises perspective image data of the second environment from a perspective of the second sensor;
apply a homography matrix including the stored, derived values to the second sensor data to warp the perspective image data of the second environment into a second ground plane;
generate a second bird's eye view of the second environment from the perspective image data of the second environment; and
provide for presentation of the second bird's eye view of the second environment on a display.

### Claim 14 (depends on 13)

The computer program product of claim 13, further comprising program code instructions to augment satellite image data of the second environment with the second bird's eye view of the second environment.

### Claim 15 (depends on 9)

The computer program product of claim 9, wherein the bird's eye view image of the environment includes dynamic objects not found in a satellite image of the environment.

### Claim 16 (depends on 9)

The computer program product of claim 9, wherein the program code instructions to retrieve a satellite image of the environment associated with the location and heading comprise program code instructions to identify the location in a satellite image map database and to identify a portion of the satellite image corresponding to the heading.

### Claim 17

A method comprising:
receiving first sensor data from a first sensor in an environment, wherein the first sensor data comprises perspective image data of the environment, wherein the first sensor data comprises a location and a heading;
retrieving a satellite image of the environment associated with the location and heading, wherein the satellite image is captured from a position above a ground plane; and
generating a bird's eye view image of the environment above the ground plane from the first sensor data by warping the first sensor data from the perspective image data of the environment based on processing of the first sensor data relative to the satellite image.

### Claim 18 (depends on 17)

The method of claim 17, wherein processing of the first sensor data relative to the satellite image comprises inputting the first sensor data and the satellite image into a deep neural network to regress the bird's eye view image.

### Claim 19 (depends on 17)

The method of claim 17, further comprising applying a Generative Adversarial Network (GAN) to the bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
deriving values of a homography matrix between the first sensor data and the stabilized bird's eye view image; and
storing the derived values of the homography matrix.
