US 10,891,735 B2Granted

Generation of synthetic high-elevation digital images from temporal sequences of high-elevation digital images

AssigneeX Development LLC

InventorsJie Yang, Cheng-en Guo, Zhiqiang Yuan, Elliott Grant, Hongxu Ma

Life of this patent

Estimating the term…

Loading patent…

Abstract

Implementations relate to detecting/replacing transient obstructions from high-elevation digital images, and/or to fusing data from high-elevation digital images having different spatial, temporal, and/or spectral resolutions. In various implementations, first and second temporal sequences of high-elevation digital images capturing a geographic area may be obtained. These temporal sequences may have different spatial, temporal, and/or spectral resolutions (or frequencies). A mapping may be generated of the pixels of the high-elevation digital images of the second temporal sequence to respective sub-pixels of the first temporal sequence. A point in time at which a synthetic high-elevation digital image of the geographic area may be selected. The synthetic high-elevation digital image may be generated for the point in time based on the mapping and other data described herein.

Claims20

Dependent claims

A method implemented using one or more processors, comprising:

obtaining a first temporal sequence of high-elevation digital images,
wherein the first temporal sequence of high-elevation digital images capture a geographic area at a first temporal frequency,
wherein each high-elevation digital image of the first temporal sequence comprises a plurality of pixels that align spatially with a respective first plurality of geographic units of the geographic area, and
wherein each high-elevation digital image of the first temporal sequence is captured at a first spatial resolution;
obtaining a second temporal sequence of high-elevation digital images,
wherein the second temporal sequence of high-elevation digital images capture the geographic area at a second temporal frequency that is less than the first temporal frequency,
wherein each high-elevation digital image of the second temporal sequence comprises a plurality of pixels that align spatially with a second plurality of geographic units of the geographic area, and
wherein each high-elevation digital image of the second temporal sequence is captured at a second spatial resolution that is greater than the first spatial resolution;
generating a mapping of the pixels of the high-elevation digital images of the second temporal sequence to respective sub-pixels of the first temporal sequence, wherein the mapping is based on spatial alignment of the geographic units of the second plurality of geographic units that underlie the pixels of the second temporal sequence with portions of the geographic units of the first plurality of geographic units that underlie the respective sub-pixels;
selecting a point in time for which a synthetic high-elevation digital image of the geographic area at the second spatial resolution will be generated;
selecting, as a low-resolution reference digital image, the high-elevation digital image from the first temporal sequence that was captured in closest temporal proximity to the point in time;
determining a first deviation of ground-truth data forming the low-resolution reference digital image from corresponding data interpolated for the point in time from the first temporal sequence of high-elevation digital images;
predicting, based on the first deviation, a second deviation of data forming the synthetic high-elevation digital image from corresponding data interpolated for the point in time from the second temporal sequence of high-elevation digital images;
and
generating the synthetic high-elevation digital image based on the mapping and the predicted second deviation.

12 more elements

A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to perform the following operations:

obtaining a first temporal sequence of high-elevation digital images,
wherein the first temporal sequence of high-elevation digital images capture a geographic area at a first temporal frequency,
wherein each high-elevation digital image of the first temporal sequence comprises a plurality of pixels that align spatially with a respective first plurality of geographic units of the geographic area, and
wherein each high-elevation digital image of the first temporal sequence is captured at a first spatial resolution;
obtaining a second temporal sequence of high-elevation digital images,
wherein the second temporal sequence of high-elevation digital images capture the geographic area at a second temporal frequency that is less than the first temporal frequency,
wherein each high-elevation digital image of the second temporal sequence comprises a plurality of pixels that align spatially with a second plurality of geographic units of the geographic area, and
wherein each high-elevation digital image of the second temporal sequence is captured at a second spatial resolution that is greater than the first spatial resolution;
generating a mapping of the pixels of the high-elevation digital images of the second temporal sequence to respective sub-pixels of the first temporal sequence, wherein the mapping is based on spatial alignment of the geographic units of the second plurality of geographic units that underlie the pixels of the second temporal sequence with portions of the geographic units of the first plurality of geographic units that underlie the respective sub-pixels;
selecting a point in time for which a synthetic high-elevation digital image of the geographic area at the second spatial resolution will be generated;
selecting, as a low-resolution reference digital image, the high-elevation digital image from the first temporal sequence that was captured in closest temporal proximity to the point in time;
determining a first deviation of ground-truth data forming the low-resolution reference digital image from corresponding data interpolated for the point in time from the first temporal sequence of high-elevation digital images;
predicting, based on the first deviation, a second deviation of data forming the synthetic high-elevation digital image from corresponding data interpolated for the point in time from the second temporal sequence of high-elevation digital images;
and
generating the synthetic high-elevation digital image based on the mapping and the predicted second deviation.

12 more elements

At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

obtaining a first temporal sequence of high-elevation digital images,
wherein the first temporal sequence of high-elevation digital images capture a geographic area at a first temporal frequency,
wherein each high-elevation digital image of the first temporal sequence comprises a plurality of pixels that align spatially with a respective first plurality of geographic units of the geographic area, and
wherein each high-elevation digital image of the first temporal sequence is captured at a first spatial resolution;
obtaining a second temporal sequence of high-elevation digital images,
wherein the second temporal sequence of high-elevation digital images capture the geographic area at a second temporal frequency that is less than the first temporal frequency,
wherein each high-elevation digital image of the second temporal sequence comprises a plurality of pixels that align spatially with a second plurality of geographic units of the geographic area, and
wherein each high-elevation digital image of the second temporal sequence is captured at a second spatial resolution that is greater than the first spatial resolution;
generating a mapping of the pixels of the high-elevation digital images of the second temporal sequence to respective sub-pixels of the first temporal sequence, wherein the mapping is based on spatial alignment of the geographic units of the second plurality of geographic units that underlie the pixels of the second temporal sequence with portions of the geographic units of the first plurality of geographic units that underlie the respective sub-pixels;
selecting a point in time for which a synthetic high-elevation digital image of the geographic area at the second spatial resolution will be generated;
selecting, as a low-resolution reference digital image, the high-elevation digital image from the first temporal sequence that was captured in closest temporal proximity to the point in time;
determining a first deviation of ground-truth data forming the low-resolution reference digital image from corresponding data interpolated for the point in time from the first temporal sequence of high-elevation digital images;
predicting, based on the first deviation, a second deviation of data forming the synthetic high-elevation digital image from corresponding data interpolated for the point in time from the second temporal sequence of high-elevation digital images;
and
generating the synthetic high-elevation digital image based on the mapping and the predicted second deviation.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/242,873Filed 8 Jan 2019 · us provisional application
Priority: 19 Oct 2018
Published: US 2020/0125822 A123 Apr 2020
Granted: 12 Jan 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification44 codes

G06T 3/4007
Scaling of whole images or parts thereof, e.g. expanding or contracting: based on interpolation, e.g. bilinear interpolation
A01D 41/127
Details of combines: control or measuring arrangements specially adapted for combines
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks
G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]