# US 2021/0240986 A1 · Augmentation of unmanned-vehicle line-of-sight

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240986 A1
- Document ID: US20210240986A1
- Status: Published application
- Kind: A1
- Application number: 16/780,282
- Filed: 2020-02-03
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: Honeywell International Inc.
- Applicants: Honeywell International Inc.
- Inventors: Prasanna Murthy Gurajapu; Bimalananda Behera; Venkata Pavan M; Shreyas Joshi; Arjun Bindu Jyothikumar
- Web page: https://patentgrep.com/patent/US20210240986A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G02B 27/0172: {characterised by optical features}
- G02B 30/40: giving the observer of a single two-dimensional [2D] image a perception of depth
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 11/65: on geographic maps
- G06V 20/20: in augmented reality scenes
- G08G 5/22: located on the ground
- G08G 5/26: Transmission of traffic-related information between aircraft and ground stations
- G08G 5/53: for cruising
- G08G 5/55: for a single aircraft
- G08G 5/57: for unmanned aircraft
- G08G 5/59: in accordance with predefined flight zones, e.g. to avoid prohibited zones
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)}
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G06T 2210/12: Bounding box
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

An augmented-reality head-mounted display is configured to display an indication of a location of an unmanned aerial vehicle with respect to the field-of-view of the head-mounted display, in order to assist a UAV operator to maintain line-of-sight with the UAV.

## Claims

### Claim 1

A device comprising:
an augmented-reality (AR) head-mounted display (HMD); and
processing circuitry configured to:
determine a location of an unmanned aerial vehicle (UAV);
determine a position and orientation of the AR HMD;
based on the location of the UAV and the position and orientation of the AR HMD, determine that the UAV is outside of a field of view of the AR HMD; and
in response to determining that the UAV is outside of the field of view of the AR HMD, present, on the AR HMD, an indication of a movement to bring the UAV within the field of view of the AR HMD.

### Claim 2 (depends on 1)

The device of claim 1, wherein the processing circuitry is further configured to:
based on the location of the UAV and the position and orientation of the AR HMD, determine that the UAV is within the field of view of the AR HMD; and
in response to determining that the UAV is within the field of view of the AR HMD, present, on the AR HMD, an indication of the location of the UAV.

### Claim 3 (depends on 2)

The device of claim 2, wherein the indication of the location of the UAV comprises a bounding box around the UAV.

### Claim 4 (depends on 1)

The device of claim 1, wherein the processing circuitry is further configured to:
determine a location of an obstacle relative to the field of view of the HMD; and
present, on the AR HMD, an indication of the location of the obstacle.

### Claim 5 (depends on 4)

The device of claim 4, wherein the processing circuitry is configured to determine a location of an obstacle by processing image data from a camera.

### Claim 6 (depends on 5)

The device of claim 5, wherein the camera is located on the HMD.

### Claim 7 (depends on 4)

The device of claim 4, wherein the obstacle comprises:
a bird;
a manned aircraft;
a second UAV; or
terrain.

### Claim 8 (depends on 4)

The device of claim 4, wherein the indication of the location of the obstacle comprises a bounding box around the location of the obstacle.

### Claim 9 (depends on 1)

The device of claim 1, the processing circuitry further configured to:
determine, based on the position and orientation of the AR HMD, a location of a controlled airspace relative to the field of view of the HMD; and
present, on the AR HMD, an indication of the location of the controlled airspace.

### Claim 10 (depends on 1)

The device of claim 1, the processing circuitry further configured to determine a field of view of a camera of the UAV; and
output for display on the AR HMD an indication of the field of view of the camera.

### Claim 11 (depends on 1)

The device of claim 1, the processing circuitry further configured to output for display on the AR HMD:
a remaining battery life of the UAV;
an altitude of the UAV;
an airspeed of the UAV;
a number of satellites in communication with the UAV; or
a compass heading of the UAV.

### Claim 12 (depends on 1)

The device of claim 1, the processing circuitry further configured to output for display on the AR HMD an alert comprising:
a low UAV battery;
a GPS-signal lost;
an obstacle-proximity warning; or
a UAV mission milestone reached.

### Claim 13 (depends on 1)

The device of claim 1, the processing circuitry further configured to:
receive, from the UAV, video data; and
output for display on the AR HMD the video data.

### Claim 14 (depends on 1)

The device of claim 1, the processing circuitry further configured to:
receive a planned flightpath of the UAV; and
output for display on the AR HMD an indication of the flightpath.

### Claim 15 (depends on 11)

The device of claim 11, the processing circuitry further configured to:
determine a loss of line-of-sight between the HMD and the UAV; and
output for display on the AR HMD an indication of the loss of line-of-sight.

### Claim 16 (depends on 1)

The device of claim 1, wherein the AR HMD comprises an opaque display screen, the HMD configured to display imagery on the display screen.

### Claim 17 (depends on 1)

The device of claim 1, wherein the AR HMD comprises a see-through holographic lens, the HMD configured to display one or more virtual elements on the lens.

### Claim 18

A method comprising:
determining a location of an unmanned aerial vehicle (UAV);
determining a position and orientation of an augmented-reality (AR) head-mounted display (HMD); and
based on the location of the UAV and the position and orientation of the AR HMD, determining whether the UAV is outside of a field of view of the AR HMD; and
in response to determining whether the UAV is outside of the field of view of the AR HMD, presenting, on the AR HMD, an indication of the location of the UAV relative to the field of view of the AR HMD.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
based on the location of the UAV and the position and orientation of the AR HMD, determining that the UAV is within the field of view of the AR HMD; and
in response to determining that the UAV is within the field of view of the AR HMD, presenting, on the AR HMD, an indication of the location of the UAV.

### Claim 20 (depends on 18)

The method of claim 18, further comprising:
based on the location of the UAV and the position and orientation of the AR HMD, determining that the UAV is outside the field of view of the AR HMD; and
in response to determining that the UAV is outside the field of view of the AR HMD, presenting, on the AR HMD, an indication of a movement to bring the UAV within the field of view of the AR HMD.