# US 2021/0241440 A1 · Deep-Learned Generation of Accurate Typical Simulator Content Via Multiple Geo-Specific Data Channels

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241440 A1
- Document ID: US20210241440A1
- Status: Published application
- Kind: A1
- Application number: 16/781,789
- Filed: 2020-02-04
- Published: 2021-08-05
- Earliest claimed: 2020-02-04
- Assignees: Rockwell Collins, Inc.
- Applicants: Rockwell Collins, Inc.
- Inventors: Daniel J. Lowe; Rishabh Kaushik
- Web page: https://patentgrep.com/patent/US20210241440A1

## CPC

- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 11/00: Two-dimensional [2D] image generation
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/10048: Infrared image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30184: Infrastructure
- G09B 9/302: {the image being transformed by computer processing, e.g. updating the image to correspond to the changing point of view}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A simulator environment is disclosed. In embodiments, the simulator environment includes graphics generation (GG) processors in communication with one or more display devices. Deep learning neural networks running on the GG processors are configured for run-time generation of photorealistic, geotypical content for display. The DL networks are trained on, and use as input, a combination of image-based input (e.g., imagery relevant to a particular geographical area) and a selection of geo-specific data sources that illustrate specific characteristics of the geographical area. Output images generated by the DL networks include additional data channels corresponding to these geo-specific data characteristics, so the generated images include geotypical representations of land use, elevation, vegetation, and other such characteristics.

## Claims

### Claim 1

A simulator environment, comprising:
one or more graphics generation (GG) processors;
one or more deep learning (DL) neural networks configured to execute on the GG processors, the one or more DL neural networks configured to:
receive one or more inputs comprising at least one input image corresponding to a location and one or more geo-specific datasets corresponding to the location, each of the one or more geo-specific datasets associated with one or more characteristics of the location;
based on the one or more inputs, generate at least one output image comprising a set of pixels, each pixel corresponding to at least one of:
an image channel associated with a color value; and
a geo-specific data channel associated with at least one characteristic of the one or more characteristics.

### Claim 2 (depends on 1)

The simulator environment of claim 1, wherein the one or more DL neural networks are configured to generate at least one evaluation of the at least one output image based on the input image, a desired output, and the one or more geo-specific datasets.

### Claim 3 (depends on 2)

The simulator environment of claim 2, wherein the at least one generator network is configured to:
generate the at least one output image based on one or more of a weight and a bias;
and
adjust at least one of the weight and the bias based on the at least one evaluation.

### Claim 4 (depends on 1)

The simulator environment of claim 1, wherein:
the one or more geo-specific datasets include geographical vector data corresponding to the location;
and
the one or more geo-specific data channels correspond to at least one of a manmade structure, a waterway, a building, a roadway, and a labeled data element.

### Claim 5 (depends on 1)

The simulator environment of claim 1, wherein the one or more geo-specific datasets include elevational data corresponding to the location.

### Claim 6 (depends on 1)

The simulator environment of claim 1, wherein the one or more geo-specific datasets include at least one of ecological data and climatic data corresponding to the location.

### Claim 7 (depends on 1)

The simulator environment of claim 1, wherein the one or more geo-specific datasets include land-use data corresponding to the location.

### Claim 8 (depends on 1)

The simulator environment of claim 1, wherein the one or more geo-specific datasets include at least one of infrared (IR) imagery and near-IR imagery corresponding to the location.

### Claim 9 (depends on 1)

The simulator environment of claim 1, wherein:
the simulator environment includes a flight simulator configured to simulate an airborne vehicle;
and
the one or more DL networks are configured for run-time generation of the at least one output image.

### Claim 10 (depends on 1)

The simulator environment of claim 1, further comprising:
at least one display device in communication with the GG processors, the display device configured to display the at least one output image.