# US 11,029,683 B2 · Systems and methods for target tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,029,683 B2
- Document ID: US11029683B2
- Status: Granted
- Kind: B2
- Application number: 15/392,979
- Filed: 2016-12-28
- Published: 2017-04-20
- Granted: 2021-06-08
- Earliest claimed: 2014-07-30
- Assignees: Sz Dji Technology Co., Ltd.
- Applicants: SZ DJI TECHNOLOGY Co., Ltd.
- Inventors: Bo Zang
- Web page: https://patentgrep.com/patent/US11029683B2

## CPC

- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- B64C 39/024: {of the remote controlled vehicle type, i.e. RPV}
- B64D 47/08: Arrangements of cameras
- G05D 1/0011: {associated with a remote control arrangement}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G05D 1/12: Target-seeking control
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06K 9/0063
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G08G 5/0069
- B64C 2201/027
- B64C 2201/108
- B64C 2201/123
- B64C 2201/127
- B64C 2201/141
- B64C 2201/146
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B64C: AEROPLANES; HELICOPTERS
- B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

The present invention provides methods and systems of tracking a target with an imaging device onboard an unmanned aerial vehicle. The method may comprise identifying one or more targets in a first image captured by the imaging device, receiving a user selection of a target to be tracked from the one or more targets in the one or more images, generating target information for the target to be tracked based on the user selection of the target, detecting a deviation of a detected position and/or a detected size of the target in a second image captured by the imaging device from the selected position and/or the selected size of the target in the first image and adjusting at least one of the UAV and the carrier, to reduce the determined deviation to maintain the target substantially within the field of view of the image device, based on the deviation.

## Claims

### Claim 1

A method of tracking a target with an imaging device onboard an unmanned aerial vehicle (UAV), the method comprising:
identifying one or more targets in a first image captured by the imaging device, the imaging device coupled to the UAV via a carrier configured to permit the imaging device to move relative to the UAV;
receiving a user selection of a target to be tracked from the one or more targets in the first image, wherein the first image is displayed on a remote controller;
generating target information for the target to be tracked based on the user selection of the target, the target information comprising:
initial information generated based on coordinates and pixel size of the target in the first image, the initial information including an initial target position of the target in the first image and an initial target size of the target in the first image; and
expected information generated based on information about the target and specifying characteristics that the target is expected to meet in images captured by the imaging device, the characteristics including a target position of the target in the images captured by the imaging device and/or a target size of the target in the images captured by the imaging device, and the expected information including:
an expected target position of the target in the images captured by the imaging device, the expected target position being away from the initial target position by a predetermined offset; and/or
an expected target size of the target in the images captured by the imaging device, the expected target size equaling the initial target size multiplied by a predetermined ratio;
detecting the target in a second image captured by the imaging device based on the initial information;
detecting a deviation of a detected position and/or a detected size of the target in the second image from the expected target position and/or the expected target size; and
adjusting at least one of the UAV or the carrier based on the deviation, to reduce the deviation to maintain the target to have a relative position and/or a relative size within a field of view of the image device that corresponds to the expected target position and/or the expected target size.

### Claim 2 (depends on 1)

The method of claim 1, wherein the deviation is related to a change in position of the target and a calculated adjustment is related to an angular velocity of the imaging device at least relative to a pitch axis of the imaging device.

### Claim 3 (depends on 2)

The method of claim 2, wherein the calculated adjustment is configured to generate control signals for the carrier so as to cause the imaging device to move relative to the UAV.

### Claim 4 (depends on 1)

The method of claim 1, further comprises transmitting, in real-time, the images captured by the imaging device to the remote controller.

### Claim 5 (depends on 1)

The method of claim 1, wherein the target information further comprises target type information comprising color, texture, or pattern information.

### Claim 6 (depends on 5)

The method of claim 5, wherein identifying the one or more targets in the first image captured by the imaging device comprises identifying, based on the target type information, using an image recognition algorithm.

### Claim 7 (depends on 1)

The method of claim 1, wherein adjusting the at least one of the UAV or the carrier based on the deviation includes adjusting the at least one of the UAV or the carrier at an adjustment speed proportional to the deviation.

### Claim 8

A system for controlling an unmanned aerial vehicle (UAV), comprising:
one or more receivers configured to receive from a remote controller one or more navigation commands to move the UAV along a flight path;
an imaging device coupled to the UAV via a carrier configured to permit the image device to move relative to the UAV, and
one or more processors configured to:
receive a user selection of a target in a first image displayed on the remote controller;
generate target information for the target to be tracked based on the user selection of the target, the target information comprising:
initial information generated based on coordinates and pixel size of the target in the first image, the initial information including an initial target position of the target in the first image and an initial target size of the target in the first image; and
expected information generated based on information about the target and specifying characteristics that the target is expected to meet in images captured by the imaging device, the characteristics including a target position of the target in the images captured by the imaging device and/or a target size of the target in the images captured by the imaging device, and the expected information including:
an expected target position of the target in the images captured by the imaging device, the expected target position being away from the initial target position by a predetermined offset; and/or
an expected target size of the target in the images captured by the imaging device, the expected target size equaling the initial target size multiplied by a predetermined ratio;
detect the target in a second image captured by the imaging device based on the initial information;
detect a deviation of a detected position and/or a detected size of the target in the second image from the expected target position and/or the expected target size; and
adjust at least one of the UAV or the carrier based on the deviation to reduce the deviation to maintain the target to have a relative position and/or a relative size within a field of view of the image device that corresponds to the expected target position and/or the expected target size.

### Claim 9 (depends on 8)

The system of claim 8, wherein the deviation is related to a change in position of the target and a calculated adjustment is related to an angular velocity of the imaging device at least relative to a pitch axis of the imaging device.

### Claim 10 (depends on 8)

The system of claim 8, wherein the one or more processors are further configured to transmit, in real-time, images captured by the imaging device to the remote controller.

### Claim 11 (depends on 8)

The system of claim 8, wherein the user selection of the target is achieved by a user selecting an area of the first image being displayed on a display, the selected area corresponding to the target.

### Claim 12 (depends on 11)

The system of claim 11, wherein the one or more processors are further configured to display, on the display, the selected target with a selection indicator in response to the user selection of the target, the selection indicator indicating that the target has been selected by the user.

### Claim 13

An unmanned aerial vehicle (UAV) with visual tracking capability, the UAV comprising:
a carrier configured to carry an imaging device;
one or more receivers configured to receive from a remote controller one or more navigation commands to move the UAV along a flight path, and
one or more processors configured to:
generate target information for the target to be tracked based on a user selection of the target, the target information comprising:
initial information generated based on coordinates and pixel size of the target in the first image, the initial information including an initial target position of the target in the first image and an initial target size of the target in the first image; and
expected information generated based on information about the target and specifying characteristics that the target is expected to meet in images captured by the imaging device, the characteristics including a target position of the target in the images captured by the imaging device and/or a target size of the target in the images captured by the imaging device, and the expected information including:
an expected target position of the target in the images captured by the imaging device, the expected target position being away from the initial target position by a predetermined offset; and/or
an expected target size of the target in the images captured by the imaging device, the expected target size equaling the initial target size multiplied by a predetermined ratio;
detect the target in a second image captured by the imaging device based on the initial information;
detect a deviation of a detected position and/or a detected size of the target in the second image from the expected target position and/or the expected target size; and
adjust at least one of the UAV or the carrier based on the deviation to reduce the deviation to maintain the target to have a relative position and/or a relative size within a field of view of the image device that corresponds to the selected position and/or the expected target size.

### Claim 14 (depends on 13)

The UAV of claim 13, wherein the deviation is related to a change in position of the target and a calculated adjustment is related to an angular velocity of the imaging device at least relative to a pitch axis of the imaging device.

### Claim 15 (depends on 13)

The UAV of claim 13, wherein the one or more processors are further configured to transmit, in real-time, images captured by the imaging device to the remote controller.

### Claim 16 (depends on 13)

The UAV of claim 13, wherein the user selection of the target is achieved by a user selecting an area of the first image being displayed on a display, the selected area corresponding to the target.