Reducing latency in head-mounted display for the remote operation of machinery
AssigneeAltec Industries, Inc.
InventorsJonathan Westin Sykes, Timothy J. Mourlam
Life of this patent
Estimating the term…
Abstract
Media, systems, and methods for reducing latency in a head-mounted display for the remote operation of machinery. A remote camera sends raw video to a graphics processing unit which processes the video and transmits the updated video to a head-mounted display with limited latency such that the control of remote devices may be improved. The image may be spherically rendered and projected onto the head-mounted display with a rectangular view. The graphics processing unit may determine a rotation matrix based on a pose of the remote camera and a pose of the head-mounted display, and the image may be projected based on the rotation matrix.
Claims20
One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by at least one processor, perform a method for reducing latency for imagery captured in a remote location and displayed in a head-mounted display, the method comprising:
- receiving, from a remote capture device controlled by a robot assembly located in the remote location, an image of the remote location;
- receiving a head-mounted display pose of the head-mounted display;
- transmitting, by the remote capture device, the image to a graphics processing unit associated with the head-mounted display;
- and
- causing the graphics processing unit to execute instructions for:
- determining, by the graphics processing unit, a rotation difference based on an image pose of the image and the head-mounted display pose;
- spherically rendering, by the graphics processing unit, the image on the head-mounted display based on the rotation difference;
- responsive to a first movement of the head-mounted display, determining, by the graphics processing unit, a new head-mounted display pose,
- wherein the first movement of the head-mounted display occurs while the remote capture device is locked from moving;
- determining, by the graphics processing unit, a new rotation difference;
- rendering, by the graphics processing unit, a new image based on the new rotation difference;
- extrapolating, by the graphics processing unit, at least one future movement of the head-mounted display;
- and
- instructing the robot assembly associated with the remote capture device to initiate a new movement corresponding to the at least one future movement.
A system for viewing a remote location, comprising:
- a remote capture device configured to capture video of the remote location;
- a robot assembly coupled to the remote capture device;
- a head-mounted display configured to control movement of the remote capture device;
- and
- a graphics processing unit associated with the head-mounted display and configured to execute computer-executable instructions comprising:
- receiving, by the graphics processing unit, a frame of the video from the remote capture device;
- determining, by the graphics processing unit, an image pose by extrapolating from two or more robot poses associated with the robot assembly;
- determining, by the graphics processing unit, a rotation difference using the image pose and a head-mounted display pose;
- spherically rendering, by the graphics processing unit, the frame of the video on the head-mounted display based on the rotation difference;
- and
- responsive to detecting a vibration at the head-mounted display, mitigating the vibration using a filter.
A method for rendering imagery of a remote location, the method comprising:
- receiving an image from a remote capture device coupled to a robot assembly disposed in the remote location;
- determining an image pose by interpolating from two or more robot poses associated with the robot assembly;
- wherein the image pose is determined using interpolation based on a first robot pose and a second robot pose from the two or more robot poses, the first robot pose preceding a time at which the image was captured and the second robot pose succeeding the time at which the image was captured;
- determining, by a graphics processing unit and based on the image pose and a head-mounted display pose of a head-mounted display, a rotation difference;
- determining, by the graphics processing unit, the first robot pose based on a plurality of prior robot poses;
- and
- rendering, by the graphics processing unit and based on the rotation difference, the image in the head-mounted display.
Dependencies are inferred from the claim text.