# US 12,181,879 B2 · System and method for supporting smooth target following

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,181,879 B2
- Document ID: US12181879B2
- Status: Granted
- Kind: B2
- Application number: 17/224,391
- Filed: 2021-04-07
- Published: 2021-07-22
- Granted: 2024-12-31
- Earliest claimed: 2015-09-15
- Assignees: Sz Dji Technology Co., Ltd.
- Applicants: Sz Dji Technology Co., Ltd.
- Inventors: Cong Zhao; Minjian Pang; Rui Li; You Zhou; Zhe Liu; Guyue Zhou
- Web page: https://patentgrep.com/patent/US12181879B2

## CPC

- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G01S 3/7864: {T.V. type tracking systems}
- G05D 1/0011: {associated with a remote control arrangement}
- G05D 1/106: {Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones}
- G05D 1/12: Target-seeking control
- G05D 1/221: Remote-control arrangements
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/13: Satellite images
- G06V 20/17: taken from planes or by drones
- H04N 23/685: {performed by mechanical compensation}
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- B64U 2101/31: for surveillance
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- B64U 2201/104: using satellite radio beacon positioning systems, e.g. GPS
- B64U 2201/20: Remote controls
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2210/12: Bounding box
- H04N 23/60: Control of cameras or camera modules
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

A method for controlling target tracking includes determining whether a tracking mode of a movable object is a manual tracking mode or an automatic tracking mode; when it is determined that the tracking mode is the manual tracking mode, determining a target for the movable object from one or more images based on a user input; when it is determined that the tracking mode is the automatic tracking mode, having the movable object determine the target automatically; and directing the movable object to track a target object using the target.

## Claims

### Claim 1

A method for controlling target tracking, comprising:
determining a target for a movable object to track in one or more images captured by the movable object;
directing the movable object to track a target object using the target; and
in response to a loss of the target in the one or more images during tracking, controlling; the movable object to redetect the target using one or more subsequent images captured by the movable object to continue tracking the target object, including:
adjusting both the movable object and an imaging device carried by the movable object in a same axis to redetect the target using the one or more subsequent images captured by the imaging device to continue tracking the target object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining a target for a movable object to track in one or more images captured by the movable object further comprises:
determining the target for the movable object based on a user input; or
having the movable object determine the target automatically.

### Claim 3 (depends on 2)

The method of claim 2, wherein the having the movable object determine the target automatically further comprises:
obtaining target type information indicative of a type of target object to be tracked; and
providing the target type information to the movable object to automatically identify the target based on the target type information.

### Claim 4 (depends on 3)

The method of claim 3, wherein the target type information is obtained via a user interface provided by a control terminal remotely coupled to the movable object via a communication link.

### Claim 5 (depends on 1)

The method of claim 1, wherein the directing the movable object to track a target object using the target further comprises:
generating control data based on the target and sensing data provided by the movable object; and
generating one or more control signals to direct the movable object to track the target object,
wherein the control data is generated by at least one of a control terminal remotely coupled to the movable object via a communication link or an onboard processing system of the movable object.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
an allocation between the control terminal and the onboard processing system is determined in real time based on a plurality of factors including environment information of the movable object.

### Claim 7 (depends on 5)

The method of claim 5, wherein:
when the movable object is navigating in a relatively complex environment, the control terminal is allocated to generate more control data than the onboard processing system of the movable object; and
when the movable object is navigating in a relatively simple environment, the control terminal is allocated to generate less control data than the onboard processing system of the movable object.

### Claim 8 (depends on 5)

The method of claim 5, wherein:
when the movable object is navigating in a relatively low altitude, the control terminal is allocated to generate more control data than the onboard processing system of the movable object; and
when the movable object is navigating in a relatively high altitude, the control terminal is allocated to generate less control data than the onboard processing system of the movable object.

### Claim 9 (depends on 5)

The method of claim 5, wherein the control data includes at least a relative distance between the movable object and the target object.

### Claim 10 (depends on 5)

The method of claim 5, wherein the control data includes at least a similarity between a feature model of the target object and features from the target.

### Claim 11 (depends on 1)

The method of claim 1, wherein controlling the movable object to redetect the target using the one or more subsequent images captured by the movable object to continue tracking the target object further includes:
in response to detecting that the movable object is in an indoor environment, controlling the movable object to use vision based tracking to redetect the target only using the one or more subsequent images captured by the movable object; or
in response to detecting that the movable object is in an outdoor environment, controlling the movable object to locate the target object using a positioning device carried by the target object for redetecting the target using one or more subsequent images captured by the movable object to continue tracking the target object.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the positioning device carried by the target object is a first positioning device; and
the controlling the movable object to locate the target object further comprises:
controlling the movable object to locate the target object according to a relative distance and/or orientation calculated based on sensor data obtained from the first positioning device carried by the target object and a second positioning device on the movable object.

### Claim 13

A system for controlling target tracking, comprising:
one or more processors; and
one or more memories coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to:
determine a target for a movable object to track in one or more images captured by the movable object;
direct the movable object to track a target object using the target; and
in response to a loss of the target in one or more images during tracking, control the movable object to redetect the target using one or more subsequent images captured by the movable object to continue tracking the target object, including:
adjusting both the movable object and an imaging device carried by the movable object in a same axis to redetect the target using the one or more subsequent images captured by the imaging device to continue tracking the target object.

### Claim 14 (depends on 13)

The system of claim 13, wherein the instructions are further executed by the one or more processors to cause the one or more processors to:
determine the target for the movable object based on a user input; or
have the movable object determine the target automatically.

### Claim 15 (depends on 14)

The system of claim 14, wherein the instructions are further executed by the one or more processors to cause the one or more processors to:
obtain target type information indicative of a type of target object to be tracked; and
provide the target type information to the movable object to automatically identify the target based on the target type information.

### Claim 16 (depends on 15)

The system of claim 15, wherein the target type information is obtained via a user interface provided by a control terminal remotely coupled to the movable object via a communication link.

### Claim 17 (depends on 13)

The system of claim 13, wherein the instructions are further executed by the one or more processors to cause the one or more processors to:
generate control data based on the target and sensing data provided by the movable object; and
generate one or more control signals to direct the movable object to track the target object,
wherein the control data is generated by at least one of the control terminal or an onboard processing system of the movable object.

### Claim 18 (depends on 17)

The system of claim 17, wherein:
when the movable object is navigating in a relatively complex environment, the control terminal is allocated to generate more control data than the onboard processing system of the movable object; and
when the movable object is navigating in a relatively simple environment, the control terminal is allocated to generate less control data than the onboard processing system of the movable object.