# US 2019/0378423 A1 · User interaction with an autonomous unmanned aerial vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378423 A1
- Document ID: US20190378423A1
- Status: Published application
- Kind: A1
- Application number: 16/439,448
- Filed: 2019-06-12
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Assignees: Skydio, Inc.
- Applicants: Skydio, Inc.
- Inventors: Abraham Galton Bachrach; Adam Parker Bry; Matthew Joseph Donahoe; Hayk Martirosyan
- Web page: https://patentgrep.com/patent/US20190378423A1

## CPC

- G08G 5/55: for a single aircraft
- B64U 20/87: Mounting of imaging devices, e.g. mounting of gimbals
- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G05D 1/106: {Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones}
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G05D 1/606: Compensating for or utilising external environmental conditions, e.g. wind or water currents (station keeping G05D1/611)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G08G 5/34: for flight plan modification
- G08G 5/57: for unmanned aircraft
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04L 67/52: specially adapted for the location of the user terminal
- B64U 10/14: with four distinct rotor axes, e.g. quadcopters
- B64U 10/25: Fixed-wing aircraft (VTOL aircraft B64U10/20)
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2219/024: Multi-user, collaborative environment
- G08G 5/32: for flight plan preparation
- G08G 5/53: for cruising
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A technique for user interaction with an autonomous unmanned aerial vehicle (UAV) is described. In an example embodiment, perception inputs from one or more sensor devices are processed to build a shared virtual environment that is representative of a physical environment. The sensor devices used to generate perception inputs can include image capture devices onboard an autonomous aerial vehicle that is in flight through the physical environment. The shared virtual environment can provide a continually updated representation of the physical environment which is accessible to multiple network-connected devices, including multiple UAVs and multiple mobile computing devices. The shared virtual environment can be used, for example, to display visual augmentations at network-connected user devices and guide autonomous navigation by the UAV.

## Claims

### Claim 1

A method comprising:
receiving, by a computer system, perception inputs generated by a sensor device associated with an autonomous aerial vehicle in flight through a physical environment;
processing, by the computer system, the perception inputs to detect a physical object that is in the physical environment;
estimating, by the computer system, a position of the detected physical object in the physical environment;
updating, by the computer system, a shared virtual environment that is representative of the physical environment to include a virtual representation of the physical object at a virtual location in the shared virtual environment that corresponds with the estimated position of the physical object in the physical environment; and
enabling, by the computer system, access to the shared virtual environment to a plurality of network-connected user devices.

### Claim 2 (depends on 1)

The method of claim 1, wherein shared virtual environment includes a computer-generated three-dimensional (3D) model of the physical environment.

### Claim 3 (depends on 1)

The method of claim 1, wherein enabling access to the shared virtual environment includes:
hosting the shared virtual environment at a server computer system that is accessible, via a computer network, to the plurality of network-connected user devices.

### Claim 4 (depends on 1)

The method of claim 1, wherein enabling access to the shared virtual environment includes:
causing display of a first view of the shared virtual environment at a first network-connected user device of the plurality of network-connected user devices;
causing display of a second view of the shared virtual environment at a network-connected second user device of the plurality of network-connected user devices;
wherein the first network-connected user device is different than the second network-connected user device; and
wherein the first view is from a different perspective than the second view.

### Claim 5 (depends on 1)

The method of claim 1, wherein enabling access to the shared virtual environment includes:
causing display of a view of the shared virtual environment at a particular network-connected user device of the plurality of network-connected user devices;
wherein the view of the shared virtual environment depicts the virtual representation of the physical object at the virtual location in the virtual environment.

### Claim 6 (depends on 5)

The method of claim 5, wherein the view is from a perspective of a virtual camera that corresponds with a perspective of the autonomous aerial vehicle in flight in the physical environment.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving, by the computer system, perception inputs generated by a second sensor device located in the physical environment;
combining, by the computer system, the perception inputs generated by the sensor device associated with the autonomous aerial vehicle with the perception inputs generated by the second sensor device to produce combined perception inputs; and
processing, by the computer system, the combined perception inputs to detect and track the physical object.

### Claim 8 (depends on 7)

The method of claim 7, wherein the sensor device associated with an autonomous aerial vehicle is of a first sensor type and the second sensor device is of a second sensor type, and wherein the first sensor type is different than the second sensor type.

### Claim 9 (depends on 7)

The method of claim 7, wherein the second sensor device is associated with a second autonomous aerial vehicle.

### Claim 10 (depends on 7)

The method of claim 7, wherein the second sensor device is associated with a mobile device.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
tracking, by the computer system, changes in the position of the physical object in the physical environment over time; and
continually updating, by the computer system, the virtual location of the virtual representation of the physical object in the shared virtual environment in real-time based on the tracked changes in position of the physical object in the physical environment.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
receiving, by the computer system, from a first network-connected user device of the plurality of network-connected user devices, a request to place a virtual object in the shared virtual environment; and
updating, by the computer system, the shared virtual environment to include the virtual object at a specified virtual location in the shared virtual environment based on the request, wherein the specified virtual location corresponds with a particular physical location in the physical environment.

### Claim 13 (depends on 12)

The method of claim 12, wherein the virtual object is viewable as a graphical augmentation via an augmented reality (AR) device when the AR device is in proximity to the particular location in the physical environment.

### Claim 14 (depends on 12)

The method of claim 12, wherein the virtual object is useable by the autonomous aerial vehicle as a waypoint for autonomous navigation through the physical environment.

### Claim 15 (depends on 1)

The method of claim 1, wherein the request includes permissions that restrict access to the virtual object in the shared virtual environment to a particular subset of the plurality of network-connected devices.

### Claim 16 (depends on 1)

The method of claim 1, wherein the shared virtual environment comprises a continually updated 3D occupancy map, and wherein the autonomous aerial vehicle uses the continually updated 3D occupancy map to autonomously maneuver through the physical environment.

### Claim 17 (depends on 1)

The method of claim 1, wherein at least one of the plurality of network-connected user devices is a mobile device.

### Claim 18 (depends on 1)

The method of claim 1, wherein at least one of the plurality of network-connected user devices is an augmented reality (AR) display device.

### Claim 19

A method for controlling image capture by an autonomous aerial vehicle, the method comprising:
receiving, by a computer system, first images of a physical environment captured by a mobile user device;
processing, by the computer system, the images to detect a physical object in the physical environment and estimate a position of the physical object in the physical environment; and
causing, by the computer system, based on the estimated position of the physical object, the autonomous aerial vehicle to maneuver to capture second images of the physical environment that include the physical object.

### Claim 20 (depends on 19)

The method of claim 19, wherein causing the autonomous aerial vehicle to maneuver to capture second images of the physical environment that include the physical object includes:
determining, by the computer system, a target position in the physical environment that is within a line of sight of the estimated position of the physical object;
generating, by the computer system, a planned trajectory to the target position; and
generating, by the computer system, control commands configured to cause the autonomous aerial vehicle to maneuver along the planned trajectory to the target position.

### Claim 21 (depends on 20)

The method of claim 20, wherein the target position corresponds with an estimated position of the mobile device in the physical environment.

### Claim 22 (depends on 19)

The method of claim 19, further comprising:
causing, by the computer system, based on the estimated position of the physical object, a gimbal mechanism associated with the autonomous aerial vehicle to adjust an orientation of an image capture device associated with the autonomous aerial vehicle such that the physical object remains in a field of view of the image capture device while capturing the second images of the physical environment.

### Claim 23 (depends on 19)

The method of claim 19, wherein the mobile device is communicatively coupled to the autonomous aerial vehicle via a wireless communication link.

### Claim 24

A method for controlling an autonomous aerial vehicle, the method comprising:
receiving, by a computer system, from a user device, a signal indicative of a user selection of a virtual navigation object located in a shared virtual environment that is representative of a physical environment, the shared virtual environment generated in part based on perception inputs from one or more sensors in the physical environment, the shared virtual environment accessible to a plurality of network-connected devices including the user device;
generating, by the computer system, a planned trajectory through the shared virtual environment relative to the virtual navigation object; and
causing, by the computer system, the autonomous aerial vehicle to maneuver through the physical environment along a path that corresponds with the planned trajectory.

### Claim 25 (depends on 24)

The method of claim 24, wherein the user selection is received at the user device via a graphical user interface that includes a rendered view of the virtual navigation object.

### Claim 26 (depends on 25)

The method of claim 25, wherein the user device is an augmented reality (AR) device and wherein the rendered view of the virtual navigation object is displayed as a visual augmentation by the AR device.

### Claim 27 (depends on 24)

The method of claim 24, further comprising:
receiving, by the computer system, from a user device, a second signal indicative of a request to place a second virtual navigation object in the shared virtual environment;
updating, by the computer system, the shared virtual environment to include the second virtual navigation object at a second virtual location in the shared virtual environment that corresponds to a second physical location in the physical environment;
generating, by the computer system, a second planned trajectory through the shared virtual environment relative to the second virtual navigation object; and
causing, by the computer system, the autonomous aerial vehicle to maneuver through the physical environment along a second path that corresponds with the second planned trajectory.

### Claim 28 (depends on 24)

The method of claim 24, wherein the shared virtual environment comprises a continually updated 3D occupancy map that includes a plurality of voxels, each of the plurality of voxels corresponding to a portion of the physical environment that is at least partially occupied by physical objects, and wherein the planned trajectory is generated so as not to intersect the voxels of the 3D occupancy map.

### Claim 29 (depends on 24)

The method of claim 24, wherein the virtual navigation object is associated with a behavioral objective, and wherein the planned trajectory is generated based on the behavioral objective associated with the virtual navigation object.

### Claim 30 (depends on 29)

The method of claim 29, wherein the behavioral objective sets parameters for guiding the behavior of the autonomous aerial vehicle when the autonomous aerial vehicle is within proximity to a physical location in the physical environment that corresponds to a virtual location of the virtual navigation object in the shared virtual environment.

### Claim 31 (depends on 29)

The method of claim 29, wherein the behavioral objective includes any of: fly over, hover, circle, capture images, or track subject.

### Claim 32

An aerial vehicle comprising:
a sensor device; and
a computer system communicatively coupled to the sensor device, the computer system configured to:
receive perception inputs generated by the sensor device while the aerial vehicle is in flight through a physical environment;
process the perception inputs to detect a physical object that is in the physical environment;
estimate a position of the detected physical object in the physical environment;
update a shared virtual environment that is representative of the physical environment to include a virtual representation of the physical object at a virtual location in the shared virtual environment that corresponds to the estimated position of the physical object in the physical environment; and
wherein the shared virtual environment is accessible to a plurality of network-connected devices.

### Claim 33 (depends on 31)

The aerial vehicle of claim 31, wherein the computer system is further configured to:
track changes in the position of the physical object in the physical environment over time; and
continually update the virtual location of the virtual representation of the physical object in the shared virtual environment in real-time based on the tracked changes in position of the physical object in the physical environment.

### Claim 34 (depends on 31)

The aerial vehicle of claim 31, wherein the shared virtual environment is continually updated based on perception inputs from two or more of the plurality of network-connected devices.

### Claim 35 (depends on 31)

The aerial vehicle of claim 31, wherein the plurality of network-connected devices includes any one or more of the aerial vehicle, a second aerial vehicle, and a mobile user device.

### Claim 36 (depends on 31)

The aerial vehicle of claim 31, wherein the shared virtual environment comprises a continually updated 3D occupancy map that includes a plurality of voxels, each of the plurality of voxels corresponding to a portion of the physical environment that is at least partially occupied by physical objects, and wherein the computer system is further configured to:
cause the aerial vehicle to autonomously maneuver through the physical environment without colliding with physical objects by generating a planned trajectory that does not intersect the plurality of voxels of the 3D occupancy map.

### Claim 37 (depends on 35)

The aerial vehicle of claim 35, wherein the shared virtual environment includes a virtual navigation object at a virtual location that corresponds to a physical location in the physical environment and wherein the planned trajectory is generated based on the virtual navigation object.

### Claim 38 (depends on 36)

The aerial vehicle of claim 36, wherein the virtual navigation object is associated with a behavioral objective, wherein the planned trajectory is generated based on the behavioral objective associated with the virtual navigation object,

### Claim 39 (depends on 37)

The aerial vehicle of claim 37, wherein the behavioral objective sets parameters for guiding the behavior of the autonomous aerial vehicle when the autonomous aerial vehicle is within proximity to the physical location in the physical environment that corresponds to the virtual location of the virtual navigation object in the shared virtual environment.
