US 2019/0058853 A1Published application

Method of generating a predictive display for tele-operation of a remotely-operated ground vehicle

AssigneeU.S. Goverment AS Represented By the Secretary of the United States Army

InventorMARK J. BRUDNAK

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for generating a predictive image to enable remote control of an unmanned ground vehicle (UGV) under conditions of long communications latency. An original digital image of the UGV environment (from a UGV-mounted camera) is projected forward such that a lower portion falls on a ground plane defined to approximate the UGV's travel surface and an upper portion falls on a far plane defined approximately normal to the ground plane and forward of the UGV. A perspective transformation technique used to generate the predicted image separates the ground plane image from the far plane image. Two separate mathematical perspective transforms are defined and applied, one for the lower (ground plane) part of the image, and one for the upper (far plane) part of the image. The resulting predicted upper and lower image are rejoined to form the complete predicted image, which may then be presented on a video screen.

Claims8

Dependent claims

A method of generating a predictive image on a video display during tele-operation of an unmanned ground vehicle (UGV), the predictive image adjusted for a delay period due to round-trip wireless communication latency between the video display and the UGV, comprising:

a) operating a camera onboard the UGV to obtain a digital image of a vehicle environment;
b) defining an initial image plane established by an initial position of the camera at an initial time;
c) defining a predicted image plane (PIP) established by a predicted camera position at a future time assuming movement of the UGV in accordance with a vehicle dynamics model as a state estimator during the delay period;
d) defining a mathematical transformation to alter the digital image to simulate a view of the camera from a perspective of the predicted camera position;
e) performing the mathematical transform on the digital image to generate the predictive image;
and
f) displaying the predictive image using the video display;
the method CHARACTERIZED IN THAT:
the above step d) of determining the mathematical transform comprises:
1) identifying a set of points on the initial image plane defining an extent of the digital image;
2) projecting a first subset of the set of points onto a ground plane to define thereon a lower projected frame (LPF), the ground plane being approximately tangent to a traction-producing means of the UGV;
3) projecting a second subset of the set of points onto a far plane to define thereon an upper projected frame (UPF), the far plane being approximately normal to the ground plane and offset from the UGV in a direction-of-movement of the UGV;
4) projecting the UPF from the far plane onto the PIP to define an upper Perspective Adjusted Frame (PAF) portion;
5) projecting the LPF from the ground plane onto the PIP to define a lower PAF portion, the upper and lower PAF portions together comprising the PAF;
6) defining a first mathematical perspective transform to map the UPF onto the upper PAF portion, and
7) defining a second mathematical perspective transform to map the LPF onto the lower PAF portion;
and
the above step d) of performing a mathematical transform on the digital image comprises:
1) performing the first mathematical perspective transform on an upper portion of the digital image to generate a predicted upper image on an upper portion of the PIP;
2) performing the second mathematical perspective transform on a lower portion of the digital image to generate a predicted lower image on a lower portion of the PIP;
and
3) rejoining the predicted upper image and the predicted lower image.

19 more elements

A method of generating a predictive image on a video display during tele-operation of an unmanned ground vehicle (UGV), the predictive image adjusted for a round-trip wireless communication delay (2D) between the video display and the UGV during vehicle movement in accordance with an assumed vehicle dynamics model as a state estimator, comprising:

a. operating a camera onboard the UGV to obtain a digital image of a vehicle environment;
b. defining an initial image plane established by an initial position of the camera at an initial time ti;
c. defining a ground plane approximately tangent to a traction-producing means of the UGV;
d. defining a far plane approximately normal to the ground plane and offset from the UGV in a direction-of-movement of the UGV;
e. defining an apparent horizon line at an intersection between the ground plane and the far plane;
f. projecting, using a mathematical mapping technique, a set of points defining an extent of the digital image (A,B,E,F) from the initial image plane, so that a first subset of the set of points (A,B) maps onto the ground plane and a second subset of the set of points (E,F) maps onto the far plane;
g. projecting the apparent horizon line onto the initial image plane and defining points (C, D) thereon where the projected apparent horizon intersects respective left and right extents of the initial image plane;
h. defining a predicted image plane established by a predicted camera position at a future time tf assuming movement of the UGV in accordance with the vehicle dynamics model and the state estimator during the delay 2D;
i. projecting, using a mathematical mapping technique, an upper projected frame (C,D,E,F) from the far plane onto the predicted image plane, and projecting a lower projected frame (A,B,C,D) from the ground plane onto the predicted image plane, the upper and lower projected frames combining to define a perspective adjusted frame (A′-F′);
j. defining a first mathematical perspective transform to map the upper projected frame (C,D,E,F) onto a corresponding upper portion of the perspective adjusted frame (C′,D′,E′,F′);
k. defining a second mathematical perspective transform to map the lower projected frame (A,B,C,D) onto a corresponding lower portion of the perspective adjusted frame (A′,B′,C′,D′);
l. performing the first mathematical perspective transform on an upper portion of the digital image to generate a predicted upper image on an upper portion of the predicted image plane;
m. performing the second mathematical perspective transform on a lower portion of the digital image to generate a predicted lower image on a lower portion of the predicted image plane;
n. rejoining the predicted lower image and the predicted upper image to create the predictive image;
and
o. displaying the predictive image using the video display.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/652,975Filed 18 Jul 2017
Published: 21 Feb 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

H04N 7/185
For receiving images from a single remote source: from a mobile camera, e.g. for remote control
G05D 1/0038
Associated with a remote control arrangement: by providing the operator with simple or augmented images from one or more cameras located onboard…
G05D 1/0044
Associated with a remote control arrangement: by providing the operator with a computer generated representation of the environment of the…
G06T 3/04
Geometric image transformations in the plane of the image: context-preserving transformations, e.g. by using an importance map