# US 2021/0171193 A1 · Unmanned aerial vehicle control method, unmanned aerial vehicle control device, and computer readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0171193 A1
- Document ID: US20210171193A1
- Status: Published application
- Kind: A1
- Application number: 16/764,708
- Filed: 2018-09-26
- Published: 2021-06-10
- Earliest claimed: 2017-11-27
- Assignees: Beijing Jingdong Shangke Information Technology Co., Ltd.; Beijing Jingdong Century Trading Co., Ltd.
- Applicants: Beijing Jingdong Shangke Information Technology Co., Ltd.; Beijing Jingdong Century Trading Co., Ltd.
- Inventors: Chunlei MEN; Yongwei ZHANG
- Web page: https://patentgrep.com/patent/US20210171193A1

## CPC

- G05D 1/0676: {specially adapted for landing}
- G05D 3/20: using a digital comparing device
- B64D 45/08: optical
- B64U 70/00: Launching, take-off or landing arrangements
- G05D 1/102: {specially adapted for vertical take-off of aircraft}
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/104: using satellite radio beacon positioning systems, e.g. GPS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

The present disclosure provides a control method of a UAV, a control device of a UAV, and a computer-readable storage medium, and relates to the technical field of UAVs. The control method of a UAV includes: determining a deviation between a vertical mapping point on the ground and a landing point of the UAV, the deviation comprising a deviation in a horizontal axis direction of a camera coordinate system and a deviation in a vertical axis direction of the camera coordinate system; and generating speed control amounts of the UAV in the horizontal axis direction and the vertical axis direction of the camera coordinate system by a controller, using the deviation in the horizontal axis direction and the deviation in the vertical axis direction.

## Claims

### Claim 1

A control method of an unmanned aerial vehicle (UAV), comprising:
determining a deviation between a vertical mapping point on the ground and a landing point of the UAV in a horizontal axis direction of a camera coordinate system and a deviation between the vertical mapping point and the landing point in a vertical axis direction of the camera coordinate system by using a vertical height of the UAV relative to the ground, coordinates of the landing point in a pixel coordinate system, an origin of an image coordinate system in the pixel coordinate system, and normalized focal lengths of the UAV camera in the horizontal axis and the vertical axis direction of the pixel coordinate system; and
generating speed control amounts of the UAV in the horizontal axis direction and the vertical axis direction of the camera coordinate system by a controller, using the deviation in the horizontal axis direction and the deviation in the vertical axis direction.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The control method of a UAV according to claim 1, wherein determining a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system comprises:
determining a correspondence between coordinates in the pixel coordinate system and coordinates in the image coordinate system;
determining a correspondence between coordinates in the image coordinate system and coordinates in the camera coordinate system according to the imaging principle of the camera; and
determining a correspondence between coordinates in the pixel coordinate system and coordinates in the camera coordinate system, and determining a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system using this correspondence.

### Claim 4 (depends on 1)

The control method of a UAV according to claim 1, wherein a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system are determined using the following formula:
{
x
=
z

(
u
-
u
0
)

/

α
x
y
=
z

(
v
-
v
0
)

/

α
y
where x represents a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system, y represents a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system, and z represents a vertical height of the UAV relative to the ground, u represents an abscissa of the landing point in the pixel coordinate system, v represents an ordinate of the landing point in the pixel coordinate system, u0 represents an abscissa of the origin of the image coordinate system in the pixel coordinate system, and v0 represents an ordinate of the origin of the image coordinate system in the pixel coordinate system, αx represents a normalized focal length of the UAV camera in the horizontal axis direction of the pixel coordinate system, and αy represents a normalized focal length of the UAV camera in the vertical axis coordinate of the pixel coordinate system.

### Claim 5 (depends on 1)

The control method of a UAV according to claim 1, wherein:
the normalized focal length of the UAV camera in the horizontal axis direction of the pixel coordinate system is determined using a ratio of the focal length of the UAV camera to a size of each pixel in the horizontal axis direction;
the normalized focal length of the UAV camera in the vertical axis direction of the pixel coordinate system is determined using a ratio of the focal length of the UAV camera to a size of each pixel in the vertical axis direction.

### Claim 6 (depends on 1)

The control method of a UAV according to claim 1, wherein the controller is a proportional-integral-differential (PID) controller, the PID controller generating a speed control amount of the UAV in the horizontal axis direction of the camera coordinate system and a speed control amount of the UAV in the vertical axis direction of the camera coordinate system using the following formula:
{
U
x

(
t
)
=
P
x

x

(
t
)
+
I
x

[
x

(
t
)
+
…
+
x

(
0
)
]
+
D
x

[
x

(
t
)
-
x

(
t
-
1
)
]
U
y

(
t
)
=
P
y

y

(
t
)
+
I
y

[
y

(
t
)
+
…
+
y

(
0
)
]
+
D
y

[
y

(
t
)
-
y

(
t
-
1
)
]
where ux(t) represents a speed control amount of the UAV in the horizontal axis direction of the camera coordinate system at time t, uy (t) represents a speed control amount of the UAV in the vertical axis direction of the camera coordinate system at time t, Px represents a proportional control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, Py represents a proportional control parameter of the PID controller in the vertical axis direction of the camera coordinate system, Ix represents a differential control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, Iy represents a differential control parameter of the PID controller in the vertical axis direction of the camera coordinate system Dx represents an integral control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, and Dy represents an integral control parameter of the PID controller in the vertical axis direction of the camera coordinate system, x(t) represents the deviation in the horizontal axis direction at time t, and y(t) represents the deviation in the vertical axis direction at time t.

### Claim at position 7

7-12. (canceled)

### Claim 13

A control device of a UAV, comprising:
a processer; and
a memory coupled to the processer and storing instructions that when executed by the processor, cause the processor to:
determine a deviation between a vertical mapping point on the ground and a landing point of the UAV in a horizontal axis direction of a camera coordinate system and a deviation between the vertical mapping point and the landing point in a vertical axis direction of the camera coordinate system by using a vertical height of the UAV relative to the ground, coordinates of the landing point in a pixel coordinate system, an origin of an image coordinate system in the pixel coordinate system, and normalized focal lengths of the UAV camera in the horizontal; and
generate speed control amounts of the UAV in the horizontal axis direction and the vertical axis direction of the camera coordinate system by a controller, using the deviation in the horizontal axis direction and the deviation in the vertical axis direction.

### Claim 14

A non-transitory computer-readable storage medium storing computer instructions which, when executed by a processor, cause a processor to:
determine a deviation between a vertical mapping point on the ground and a landing point of the UAV in a horizontal axis direction of a camera coordinate system and a deviation between the vertical mapping point and the landing point in a vertical axis direction of the camera coordinate system by using a vertical height of the UAV relative to the ground, coordinates of the landing point in a pixel coordinate system, an origin of an image coordinate system in the pixel coordinate system, and normalized focal lengths of the UAV camera in the horizontal; and
generate speed control amounts of the UAV in the horizontal axis direction and the vertical axis direction of the camera coordinate system by a controller, using the deviation in the horizontal axis direction and the deviation in the vertical axis direction.

### Claim 15 (depends on 13)

The control device of a UAV according to claim 13, wherein determining a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system comprises:
determining a correspondence between coordinates in the pixel coordinate system and coordinates in the image coordinate system;
determining a correspondence between coordinates in the image coordinate system and coordinates in the camera coordinate system according to the imaging principle of the camera; and
determining a correspondence between coordinates in the pixel coordinate system and coordinates in the camera coordinate system, and determining a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system using this correspondence.

### Claim 16 (depends on 13)

The control device of a UAV according to claim 13, wherein a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system are determined using the following formula:
{
x
=
z

(
u
-
u
0
)

/

α
x
y
=
z

(
v
-
v
0
)

/

α
y
where x represents a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system, y represents a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system, and z represents a vertical height of the UAV relative to the ground, u represents an abscissa of the landing point in the pixel coordinate system, v represents an ordinate of the landing point in the pixel coordinate system, u0 represents an abscissa of the origin of the image coordinate system in the pixel coordinate system, and v0 represents an ordinate of the origin of the image coordinate system in the pixel coordinate system, αx represents a normalized focal length of the UAV camera in the horizontal axis direction of the pixel coordinate system, and αy represents a normalized focal length of the UAV camera in the vertical axis coordinate of the pixel coordinate system.

### Claim 17 (depends on 13)

The control device of a UAV according to claim 13, wherein:
the normalized focal length of the UAV camera in the horizontal axis direction of the pixel coordinate system is determined using a ratio of the focal length of the UAV camera to a size of each pixel in the horizontal axis direction;
the normalized focal length of the UAV camera in the vertical axis direction of the pixel coordinate system is determined using a ratio of the focal length of the UAV camera to a size of each pixel in the vertical axis direction.

### Claim 18 (depends on 13)

The control device of a UAV according to claim 13, wherein the controller is a proportional-integral-differential (PID) controller, the PID controller generating a speed control amount of the UAV in the horizontal axis direction of the camera coordinate system and a speed control amount of the UAV in the vertical axis direction of the camera coordinate system using the following formula:
{
U
x

(
t
)
=
P
x

x

(
t
)
+
I
x

[
x

(
t
)
+
…
+
x

(
0
)
]
+
D
x

[
x

(
t
)
-
x

(
t
-
1
)
]
U
y

(
t
)
=
P
y

y

(
t
)
+
I
y

[
y

(
t
)
+
…
+
y

(
0
)
]
+
D
y

[
y

(
t
)
-
y

(
t
-
1
)
]
where ux(t) represents a speed control amount of the UAV in the horizontal axis direction of the camera coordinate system at time t, uy (t) represents a speed control amount of the UAV in the vertical axis direction of the camera coordinate system at time t, Px represents a proportional control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, Py represents a proportional control parameter of the PID controller in the vertical axis direction of the camera coordinate system, Ix represents a differential control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, Iy represents a differential control parameter of the PID controller in the vertical axis direction of the camera coordinate system Dx represents an integral control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, and Dy represents an integral control parameter of the PID controller in the vertical axis direction of the camera coordinate system, x(t) represents the deviation in the horizontal axis direction at time t, and y(t) represents the deviation in the vertical axis direction at time t.

### Claim 19 (depends on 14)

The non-transitory computer readable storage medium according to claim 14, wherein determining a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system by using a vertical height of the UAV relative to the ground, coordinates of the landing point in a pixel coordinate system, an origin of the image coordinate system in the pixel coordinate system, and normalized focal lengths of the UAV camera in the horizontal axis and the vertical axis direction of the pixel coordinate system.

### Claim 20 (depends on 14)

The non-transitory computer readable storage medium according to claim 14, wherein a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system and a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system are determined using the following formula:
{
x
=
z

(
u
-
u
0
)

/

α
x
y
=
z

(
v
-
v
0
)

/

α
y
where x represents a deviation between the vertical mapping point and the landing point in the horizontal axis direction of the camera coordinate system, y represents a deviation between the vertical mapping point and the landing point in the vertical axis direction of the camera coordinate system, and z represents a vertical height of the UAV relative to the ground, u represents an abscissa of the landing point in the pixel coordinate system, v represents an ordinate of the landing point in the pixel coordinate system, u0 represents an abscissa of the origin of the image coordinate system in the pixel coordinate system, and v0 represents an ordinate of the origin of the image coordinate system in the pixel coordinate system, αx represents a normalized focal length of the UAV camera in the horizontal axis direction of the pixel coordinate system, and αy represents a normalized focal length of the UAV camera in the vertical axis coordinate of the pixel coordinate system.

### Claim 21 (depends on 14)

The non-transitory computer readable storage medium according to claim 14, wherein:
the normalized focal length of the UAV camera in the horizontal axis direction of the pixel coordinate system is determined using a ratio of the focal length of the UAV camera to a size of each pixel in the horizontal axis direction;
the normalized focal length of the UAV camera in the vertical axis direction of the pixel coordinate system is determined using a ratio of the focal length of the UAV camera to a size of each pixel in the vertical axis direction.

### Claim 22 (depends on 14)

The non-transitory computer readable storage medium according to claim 14, wherein the controller is a proportional-integral-differential (PID) controller, the PID controller generating a speed control amount of the UAV in the horizontal axis direction of the camera coordinate system and a speed control amount of the UAV in the vertical axis direction of the camera coordinate system using the following formula:
{
U
x

(
t
)
=
P
x

x

(
t
)
+
I
x

[
x

(
t
)
+
…
+
x

(
0
)
]
+
D
x

[
x

(
t
)
-
x

(
t
-
1
)
]
U
y

(
t
)
=
P
y

y

(
t
)
+
I
y

[
y

(
t
)
+
…
+
y

(
0
)
]
+
D
y

[
y

(
t
)
-
y

(
t
-
1
)
]
where ux(t) represents a speed control amount of the UAV in the horizontal axis direction of the camera coordinate system at time t, uy (t) represents a speed control amount of the UAV in the vertical axis direction of the camera coordinate system at time t, P, represents a proportional control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, Py represents a proportional control parameter of the PID controller in the vertical axis direction of the camera coordinate system, Ix represents a differential control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, Iy represents a differential control parameter of the PID controller in the vertical axis direction of the camera coordinate system Dx represents an integral control parameter of the PID controller in the horizontal axis direction of the camera coordinate system, and Dy represents an integral control parameter of the PID controller in the vertical axis direction of the camera coordinate system, x(t) represents the deviation in the horizontal axis direction at time t, and y(t) represents the deviation in the vertical axis direction at time t.
