US 10,798,343 B2Granted

Augmented video system providing enhanced situational awareness

AssigneeInsitu, Inc.

InventorsDarcy Davidson, Jr., Theodore T. Trowbridge

Life of this patent

Estimating the term…

Loading patent…

Abstract

A facility, comprising systems and methods, for providing enhanced situational awareness to captured image data is disclosed. The disclosed techniques are used in conjunction with image data, such as a real-time or near real-time image stream captured by a camera attached to an unmanned system, previously captured image data, rendered image data, etc. The facility enhances situational awareness by projecting overlays onto captured video data or “wrapping” captured image data with previously-captured and/or “synthetic world” information, such as satellite images, computer-generated images, wire models, textured surfaces, and so on. The facility also provides enhanced zoom techniques that allow a user to quickly zoom in on an object or area of interest using a combined optical and digital zoom technique. Additionally, the facility provides a digital lead indicator designed to reduce operator-induced oscillations in commanding an image capturing device.

Claims19

Dependent claims

A method for providing situational awareness to image data captured by an image capturing device carried by an unmanned aircraft, the method comprising:

determining, by executing instructions with at least one processor, a transformation matrix based on metadata associated with at least one of a plurality of frames of the captured image data;
for at least one overlay, applying, by executing instructions with the at least one processor, the transformation matrix to the at least one overlay to transform the at least one overlay;
and
providing, by executing instructions with the at least one processor, the at least one transformed overlay for display;
and
in response to receiving a request to change the orientation of the image capturing device,
determining, by executing instructions with the at least one processor, a horizontal field of view for the image capturing device,
determining, by executing instructions with the at least one processor, a vertical field of view for the image capturing device,
determining, by executing instructions with the at least one processor, a horizontal slew rate for changing the orientation of the image capturing device,
determining, by executing instructions with the at least one processor, a vertical slew rate for changing the orientation of the image capturing device,
determining, by executing instructions with the at least one processor, a width of a displayed image,
determining, by executing instructions with the at least one processor, a height of the displayed image,
determining, by executing instructions with the at least one processor, an advance period,
calculating, by executing instructions with the at least one processor, a coordinate based on the horizontal field of view, the vertical field of view, the horizontal slew rate, the vertical slew rate, the width, the height, and the advance period, and
providing, by executing instructions with the at least one processor, an indication of a digital lead indicator at the calculated coordinate for display.

12 more elements

A non-transitory machine readable medium comprising instructions, which when executed, cause a processor to at least:

determine a transformation matrix based on metadata associated with at least one of a plurality of frames of image data captured by an image capturing device carried by an unmanned aircraft;
for at least one overlay, apply the transformation matrix to the at least one overlay to transform the at least one overlay;
provide the at least one transformed overlay for display;
and
in response to receiving a request to change the orientation of the image capturing device,
determine a horizontal field of view for the image capturing device,
determine a vertical field of view for the image capturing device,
determine a horizontal slew rate for changing the orientation of the image capturing device,
determine a vertical slew rate for changing the orientation of the image capturing device,
determine a width of a displayed image,
determine a height of the displayed image,
determine an advance period,
calculate a coordinate based on the horizontal field of view, the vertical field of view, the horizontal slew rate, the vertical slew rate, the width, the height, and the advance period, and
provide an indication of a digital lead indicator at the calculated coordinate for display.

11 more elements

An apparatus for providing situational awareness to image data captured by an image capturing device carried by an unmanned aircraft system, the apparatus comprising:

a render hybrid data component to:
determine a transformation matrix based on metadata associated with at least one of a plurality of frames of the captured image data,
for at least one overlay, apply the transformation matrix to the at least one overlay to transform the at least one overlay, and
provide the at least one transformed overlay for display;
and
a digital lead indicator component, in response to receiving a request to change the orientation of the image capturing device, to:
determine a horizontal field of view for the image capturing device,
determine a vertical field of view for the image capturing device,
determine a horizontal slew rate for changing the orientation of the image capturing device,
determine a vertical slew rate for changing the orientation of the image capturing device,
determine a width of a displayed image,
determine a height of the displayed image,
determine an advance period,
calculate a coordinate based on the horizontal field of view, the vertical field of view, the horizontal slew rate, the vertical slew rate, the width, the height, and the advance period, and
provide an indication of a digital lead indicator at the calculated coordinate for display.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/408,073Filed 9 May 2019 · continuation
Priority: 30 Jan 2013
Published: US 2019/0349553 A114 Nov 2019
Granted: 6 Oct 2020
Term ends: Not estimatedThe source records don’t support an estimate.

Classification13 codes

G06T 17/05
Three-dimensional [3D] modelling for computer graphics: geographic models
G01C 11/00
Photogrammetry or videogrammetry
G01C 11/02
Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying: picture taking arrangements specially adapted for photogrammetry…
G05D 1/0038
Associated with a remote control arrangement: by providing the operator with simple or augmented images from one or more cameras located onboard…

Family

Application13/754,795Parent of this continuation · filed 30 Jan 2013
Application15/162,463Parent of this continuation · filed 23 May 2016
US 9,380,275 B2Parent of this continuation · filed 28 Jun 2016
US 2019/0349553 A1Pre-grant publication · published 14 Nov 2019
US 10,334,210 B2Parent of this continuation
US 2014/0267723 A1Published application
US 2016/0344980 A1Published application
US 10,798,343 B2This patent

From the US filing history. Foreign family members aren’t listed.