# US 2019/0377345 A1 · Fitness and sports applications for an autonomous unmanned aerial vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377345 A1
- Document ID: US20190377345A1
- Status: Published application
- Kind: A1
- Application number: 16/439,504
- Filed: 2019-06-12
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Assignees: Skydio, Inc.
- Applicants: Skydio, Inc.
- Inventors: Abraham Galton Bachrach; Adam Parker Bry; Matthew Joseph Donahoe; Hayk Martirosyan; Tom Moss
- Web page: https://patentgrep.com/patent/US20190377345A1

## CPC

- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- B64U 20/87: Mounting of imaging devices, e.g. mounting of gimbals
- G05D 1/689: Pointing payloads towards fixed or moving targets (positioning towed, pushed or suspended implements G05D1/672)
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 20/13: Satellite images
- G06V 20/17: taken from planes or by drones
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- B64U 10/14: with four distinct rotor axes, e.g. quadcopters
- B64U 10/25: Fixed-wing aircraft (VTOL aircraft B64U10/20)
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- G06T 2207/30224: Ball; Puck
- G06T 2207/30228: Playing field
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Sports and fitness applications for an autonomous unmanned aerial vehicle (UAV) are described. In an example embodiment, a UAV can be configured to track a human subject using perception inputs from one or more onboard sensors. The perception inputs can be utilized to generate values for various performance metrics associated with the activity of the human subject. In some embodiments, the perception inputs can be utilized to autonomously maneuver the UAV to lead the human subject to satisfy a performance goal. The UAV can also be configured to autonomously capture images of a sporting event and/or make rule determinations while officiating a sporting event.

## Claims

### Claim 1

A method for facilitating fitness training using an unmanned aerial vehicle (UAV), the method comprising:
receiving, by a computer system, a behavioral objective input, the behavioral objective input configured based on a performance goal for a human subject;
receiving, by a computer system, perception inputs generated by one or more sensors associated with the UAV;
generating, by the computer system, a planned trajectory through a physical environment based on the behavioral objective input and the perception inputs, the planned trajectory configured to cause the UAV to lead the human subject so as to satisfy the performance goal; and
causing, by the computer system, the UAV to autonomously maneuver along the planned trajectory.

### Claim 2 (depends on 1)

The method of claim 1, wherein the performance goal defines a particular running pace and wherein causing the UAV to maneuver along the planned trajectory includes causing the UAV to fly at a speed that sets the particular pace for the human subject.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the planned trajectory includes:
processing, by computer system, the behavioral objective with one or more other behavioral objectives using any of gradient-based optimization, gradient-free optimization, sampling, or end-to-end.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining, by the computer system, based on the perception inputs, that the human subject is injured and/or that the human subject is tired; and
adjusting, by the computer system, the planned trajectory in response to determining that the human subject is injured and/or that the human subject is tired.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining, by the computer system, based on the perception inputs, a terrain characteristic of the physical environment and adjusting the planned trajectory in response to determining the terrain characteristic of the physical environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein the behavioral objective input is based on tracking data generated by a second UAV tracking a second human subject, the tracking data indicative of the motion of the second human subject.

### Claim 7 (depends on 6)

The method of claim 6, wherein the planned trajectory corresponds with the motion of the second human subject.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
generating, by the computer system, based on the perception inputs, tracking data indicative of the motion of the human subject; and
transmitting, by the computer system, via a wireless communication link, the tracking data to a second UAV;
wherein the tracking data is utilized by the second UAV to autonomously maneuver along a second planned trajectory that corresponds with the motion of the human subject.

### Claim 9 (depends on 1)

The method of claim 1, wherein the perception inputs include images captured by a camera coupled to the UAV.

### Claim 10 (depends on 1)

The method of claim 1, wherein the performance goal is any of a particular pace, a particular speed, a particular time, or a particular distance.

### Claim 11 (depends on 1)

The method of claim 1, wherein the human subject is any of a runner, a swimmer, a bicyclist, a skier, or a snowboarder.

### Claim 12

A method for facilitating fitness training using an unmanned aerial vehicle (UAV), the method comprising:
receiving, by a computer system, images of a physical environment captured by one or more image capture devices associated with the UAV, the UAV in autonomous flight through the physical environment in proximity to a human subject;
processing, by the computer system, the received images to detect and track a motion of the human subject through the physical environment;
analyzing, by the computer, system, the motion of the human subject based on the tracking;
generating, by the computer system, based on the analysis, a value for a performance metric associated with the motion of the human subject; and
causing display, by the computer system, of a visual output that includes at least some of the images of the physical environment and an indication of the value of the performance metric.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
causing display, by the computer system, of the visual output at a mobile device.

### Claim 14 (depends on 13)

The method of claim 13, wherein the mobile device is communicatively coupled to the UAV via a wireless communication link and wherein the visual output is displayed at the mobile device in real-time as the UAV is in autonomous flight and tracking the human subject.

### Claim 15 (depends on 12)

The method of claim 12, further comprising:
generating, by the computer system, a graphical element based on the analysis of the motion of the human subject;
wherein the visual output includes the graphical element.

### Claim 16 (depends on 15)

The method of claim 15, wherein the graphical element includes any of:
a graphical representation of a trajectory of the human subject; or
a graphical representation of a skeletal structure of the human subject.

### Claim 17 (depends on 12)

The method of claim 12, wherein the performance metric includes any of speed, total run time, lap time, gait, pace, or elevation gain.

### Claim 18 (depends on 12)

The method of claim 12, wherein the human subject is any of a runner, a swimmer, a bicyclist, a skier, or a snowboarder.

### Claim 19

A method for capturing images of a sporting event using an unmanned aerial vehicle (UAV), the method comprising:
receiving, by a computer system, perception inputs generated by one or more sensors associated with the UAV, the one or more sensors including an image capture device;
detecting and tracking, by the computer system, based on the perception inputs, a moving area of interest within a particular area associated with the sporting event; and
causing, by the computer systems, the UAV to autonomously maneuver and adjust an orientation of the image capture device to keep the area of interest in a field of view of the image capture device.

### Claim 20 (depends on 19)

The method of claim 19, wherein the particular area associated with the sporting event is a field of play.

### Claim 21 (depends on 19)

The method of claim 19, wherein the moving area of interest corresponds with the motion of a ball in play.

### Claim 22 (depends on 19)

The method of claim 19, wherein causing the UAV to autonomously maneuver includes:
generating and continually updating, by the computer system, based on the perception inputs, a planned trajectory configured to keep the UAV within a threshold proximity of the moving area of interest while simultaneously avoiding overflying the particular area of the sporting event.

### Claim 23 (depends on 19)

The method of claim 19, further comprising:
causing display, by the computer system, of images from the image capture device at a display device.

### Claim 24 (depends on 19)

The method of claim 19, further comprising:
accessing, by the computer system, a rule associated with the sporting event; and
processing, by the computer system, the perception inputs to detect an activity occurring during the sporting event; and
applying, by the computer system, the accessed rule to the detected activity to generate a rule determination.

### Claim 25 (depends on 24)

The method of claim 24, further comprising:
causing, by the computer system, the UAV to generate an audible output indicative of the rule determination; and/or
causing, by the computer system, a public address system at a venue hosting the sporting event to generate an audible output indicative of the rule determination.

### Claim 26 (depends on 24)

The method of claim 24, further comprising:
receiving, by the computers system, a plurality of other rule determinations from a plurality of other UAVs, each of the plurality of other rule determinations based on independent application of the rule to activity identified by a different one of the plurality of other UAVs; and
generating, by the computer system, a final rule determination if the rule determination and the plurality of other rule determinations satisfy a specified matching criterion.

### Claim 27 (depends on 24)

The method of claim 24, wherein accessing the rule associated with the sporting event includes:
processing, by the computer system, the perception inputs to identify a characteristic of the sporting event; and
selecting, by the computer system, based on the identified characteristic, the rule from a library including a plurality of rules for a plurality of types of sporting events.

### Claim 28 (depends on 24)

The method of claim 24, further comprising:
causing display, by the computer system, of a visual output at a display device, the visual output including images from the image capture device and an indication of the rule determinization.

### Claim 29

An unmanned aerial vehicle (UAV) comprising:
a propulsion system;
a sensor device; and
a computer system communicatively coupled to the propulsion system and sensor device, the computer system configured to:
receive a behavioral objective input, the behavioral objective input configured based on a performance goal for a human subject;
receive perception inputs generated by the sensor device;
generate a planned trajectory through the physical environment based on the behavioral objective input and the perception inputs, the planned trajectory configured to cause the UAV to lead the human subject so as to satisfy the performance goal; and
control the propulsion system to autonomously maneuver the UAV along the planned trajectory.

### Claim 30 (depends on 29)

The UAV of claim 29, wherein the computer system is further configured to:
process the received perception inputs to detect and track a motion of the human subject through the physical environment;
analyze the motion of the human subject based on the tracking;
generate, based on the analysis, a value for a performance metric associated with the motion of the human subject; and
cause display of a visual output that includes the value for the performance metric.

### Claim 31

An unmanned aerial vehicle (UAV) comprising:
a propulsion system;
a sensor device;
a gimbal mechanism;
a camera rotatably coupled to a body of the UAV via the gimbal mechanism;
a computer system communicatively coupled to the propulsion system, sensor device, and gimbal mechanism, the computer system configured to:
receive perception inputs generated by the sensor device;
detect and track, based on the perception inputs, a moving area of interest within a particular area associated with a sporting event; and
control the propulsion system gimbal mechanism to autonomously maneuver the UAV and rotate the camera relative to the body of the UAV so as to keep the area of interest in a field of view of the camera.

### Claim 32 (depends on 31)

The UAV of claim 31 further comprising:
a storage device, the storage device storing a library of a plurality of rules associated with a plurality of different types of sporting events;
wherein the computer system is further configured to:
process the perception inputs to identify a characteristic of the sporting event and detect and activity occurring during the sporting event;
select, based on the identified characteristic, a particular rule from the library of the plurality of rules;
apply the particular rule to the detected activity to generate a rule determination; and
generate an output indicative of the rule determination.