# US 2016/0259330 A1 · Point-and-Click Control of Unmanned, Autonomous Vehicle Using Omni-Directional Visors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259330 A1
- Document ID: US20160259330A1
- Status: Published application
- Kind: A1
- Application number: 14/989,599
- Filed: 2016-01-06
- Published: 2016-09-08
- Earliest claimed: 2015-03-06
- Assignees: Robotic Research, LLC
- Inventors: Alberto Daniel Lacaze; Karl Nicholas Murphy
- Web page: https://patentgrep.com/patent/US20160259330A1

## CPC

- G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)}
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- G08C 17/02: using a radio link
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G)

## Abstract

The proposed method outlines a new control mechanism well-suited for small, unmanned aerial vehicles traversing in a GPS-denied areas. It has the strong advantage of simplifying the interface, so that even an untrained operator can handle the difficult, dynamic problems encountered in closed quarters. The proposed system seamlessly integrates point-and-click control with way-point navigation, in an intuitive interface. An additional advantage of the proposed system is that it adds minimal hardware to the payload of the UAV, and can possibly, strongly diminish the bandwidth and delay effects of the communication channel.

## Claims

### Claim 1

A control system device comprising:
a teleoperated vehicle;
an omnidirectional camera or wide field of view mounted on the vehicle;
a 3D display worn by the operator to visualize video or a sequence of images collected by the vehicle;
a communication mechanism between the vehicle and the operator; and
a display overlay that indicates to the operator the desired goal location of the vehicle.

### Claim 2 (depends on 1)

The device of claim 1, wherein
the 3D stereo goggles are further comprised of an embedded IMU; and
wherein the pose of the head of the operator can be used to provide “virtual reality”.

### Claim 3 (depends on 1)

The device of claim 1, further comprising a range sensor that covers the direction of travel as to stop the vehicle from colliding to the location assigned by the operator.

### Claim 4 (depends on 2)

The device of claim 2, wherein the range sensor is either acoustic or LADAR.

### Claim 5 (depends on 1)

The device of claim 1, further comprising of a mechanism for rotating the images displayed to the operator to account for the operator's self motion.

### Claim 6 (depends on 1)

The device of claim 1, further comprising a mechanism for rotation the images displayed to the operator to account for the vehicles motion.

### Claim 7 (depends on 1)

The device of claim 1, further comprising adding the distance to collision, and end estimated time of arrival in the direction indicated by the operator.

### Claim 8 (depends on 1)

The device of claim 1, wherein the overlays are displayed three dimensionally by the stereo goggles to express the trajectory and assigned stopping location.

### Claim 9 (depends on 8)

The device of claim 8, wherein a model of motion of the vehicle is used to predict the trajectory being expressed on the display.

### Claim 10 (depends on 1)

The device of claim 1, further comprising an input device allowing the operator to select the distance along the trajectory displayed in the stereo goggles.

### Claim 11 (depends on 1)

The device of claim 1, further comprising the displaying a rendering of the representation of the vehicle.

### Claim 12 (depends on 1)

The device of claim 1, wherein the cameras on the vehicle provide stereo omnidirectional imagery or generate stereo image pairs by rotating a camera

### Claim 13 (depends on 1)

The device of claim 1, wherein the obstacles sensed by the vehicle are also overlaid on the stereo goggles or other display.

### Claim 14 (depends on 1)

The device of claim 1, wherein the Euler angles of the display stereo goggles are used to steer the goal location or the trajectory of the vehicle.

### Claim 15 (depends on 1)

The device of claim 1, wherein multiple operators can view the imagery and select different goal points.

### Claim 16 (depends on 1)

The device of claim 1, wherein a 2D or 3D top view map showing the location and trajectory of the vehicle is displayed either as an overlay or utilizing the areas in the hemisphere where the cameras do not cover.

### Claim 17 (depends on 1)

The device of claim 1, wherein the operator can rewind the imagery to a previous time and still select the goal point.

### Claim 18 (depends on 1)

The device of claim 1, wherein a top view display of the obstacles found by the vehicle are displayed on the 3D stereo goggles.

### Claim 19 (depends on 1)

The device of claim 1, further comprising a navigation unit that controls or guides the vehicle to the assigned location.

### Claim 20 (depends on 1)

The device of claim 1, further comprising a depiction of the final pose of the system and a method for selecting this final pose or intermediate poses.

### Claim 21

A control system device comprising:
an unmanned air or ground vehicle;
an omnidirectional camera or wide field of view mounted on the unmanned vehicle;
a set of 3D stereo goggles that displays a series of images collected on the unmanned vehicle;
a communication mechanism between the vehicle and the operator; and
a display overlay that indicates to the operator the desired goal location of the vehicle.
