# US 12,008,824 B2 · Target positioning method and device, and unmanned aerial vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,824 B2
- Document ID: US12008824B2
- Status: Granted
- Kind: B2
- Application number: 17/357,346
- Filed: 2021-06-24
- Published: 2021-10-14
- Granted: 2024-06-11
- Earliest claimed: 2018-12-24
- Assignees: Autel Robotics Co., Ltd.
- Applicants: Autel Robotics Co., Ltd.
- Inventors: Xin Zheng
- Web page: https://patentgrep.com/patent/US12008824B2

## CPC

- G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces}
- B64U 20/87: Mounting of imaging devices, e.g. mounting of gimbals
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/292: Multi-camera tracking
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 10/757: {Matching configurations of points or features}
- G06V 20/13: Satellite images
- B64U 2101/30: for imaging, photography or videography
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20132: Image cropping
- G06T 2207/30196: Human being; Person
- G06T 2207/30241: Trajectory
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present invention relates to a target positioning method and device, and an unmanned aerial vehicle. The method includes: acquiring a target image through an image acquisition device; acquiring the position information of a target in the target image according to the target image; acquiring original point clouds of an environment in front of the unmanned aerial vehicle through a depth sensor; acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and acquiring the position information of the target in a three-dimensional space according to the point clouds corresponding to the target.

## Claims

### Claim 1

A target positioning method, applied to an unmanned aerial vehicle, the unmanned aerial vehicle comprising an image acquisition device and a depth sensor, and the method comprising:
acquiring a target image through the image acquisition device;
acquiring the position information of a target in the target image according to the target image;
acquiring original point clouds of an environment in front of the unmanned aerial vehicle through the depth sensor;
acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and
acquiring the position information of the target in a three-dimensional space according to the point clouds corresponding to the target;
wherein the position information of the target in the target image comprises minimum bounding box coordinates of the target in the target image, and acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image comprises:
projecting the original point clouds to a plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located; and
determining the original point clouds of which the coordinates fall into a minimum bounding box in the original point clouds as the point clouds corresponding to the target.

### Claim 2 (depends on 1)

The target positioning method according to claim 1, wherein projecting the original point clouds into the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located comprises:
transforming the coordinates of the original point clouds in a depth sensor coordinate system into the coordinates of the original point clouds in an image acquisition device coordinate system; and
projecting the coordinates of the original point clouds in the image acquisition device coordinate system to the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located.

### Claim 3 (depends on 2)

The target positioning method according to claim 2, wherein the coordinates of the original point clouds in the image acquisition device coordinate system satisfy:
(
x
′
y
′
z
′
)
=
T
×
(
x
y
z
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, (x, y, z) are the coordinates of the original point clouds in the depth sensor coordinate system, and T is a transformation matrix from the depth sensor coordinate system to the image acquisition device coordinate system.

### Claim 4 (depends on 3)

The target positioning method according to claim 3, wherein the transformation matrix T from the depth sensor coordinate system to the image acquisition device coordinate system satisfies:
T=T2T×T1,
wherein T1 is a transformation matrix from the depth sensor coordinate system to an unmanned aerial vehicle fuselage coordinate system, and T2 is a transformation matrix from the image acquisition device coordinate system to the unmanned aerial vehicle fuselage coordinate system.

### Claim 5 (depends on 1)

The target positioning method according to claim 1, wherein a coordinates (u, v) of the original point clouds in the plane where the target image is located satisfy:
d
×
(
u
v
)
=
k
×
(
x
′
y
′
z
′
)
⁢
⁢
and
k
=
(
f
x
0
c
x
0
f
y
c
y
0
0
1
)
⁢
⁢
and
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, d is the depth of the original point clouds, and k is an intrinsic matrix of the image acquisition device.

### Claim 6 (depends on 1)

The target positioning method according to claim 1, wherein before acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image, the method further comprises:
filtering the original point clouds of which the coordinates fall into the minimum bounding box so as to remove the point clouds which do not belong to the target.

### Claim 7 (depends on 1)

The target positioning method according to claim 1, wherein acquiring the position information of the target in the three-dimensional space according to the point clouds corresponding to the target comprises:
establishing a three-dimensional map of the environment in front of the unmanned aerial vehicle according to the original point clouds of the environment in front of the unmanned aerial vehicle; and
taking an average value of the three-dimensional coordinates of the point clouds corresponding to the target, wherein the average value is configured to represent the position of the target in the three-dimensional map.

### Claim 8 (depends on 1)

The target positioning method according to claim 1, wherein the depth sensor comprises at least one of a binocular camera, a structured light camera, a time of flight (TOF) camera, and a laser radar.

### Claim 9

A target positioning device, applied to an unmanned aerial vehicle, the unmanned aerial vehicle comprising an image acquisition device and a depth sensor, and the target positioning device comprising:
a processor configured to acquire a target image collected by an image acquisition device and acquire the position information of a target in the target image according to the target image;
acquire original point clouds of an environment in front of the unmanned aerial vehicle collected through the depth sensor;
acquire the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and
acquire the position information of the target in a three-dimensional space according to the point clouds corresponding to the target;
project the original point clouds to a plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located; and
determine the original point clouds of which the coordinates fall into a minimum bounding box in the original point clouds as the point clouds corresponding to the target.

### Claim 10 (depends on 9)

The device according to claim 9, wherein the processor is further configured to
transform the coordinates of the original point clouds in a depth sensor coordinate system into the coordinates of the original point clouds in an image acquisition device coordinate system; and
project the coordinates of the original point clouds in the image acquisition device coordinate system to the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located.

### Claim 11 (depends on 10)

The device according to claim 10, wherein the coordinates of the original point clouds in the image acquisition device coordinate system satisfy:
(
x
′
y
′
z
′
)
=
T
×
(
x
y
z
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, (x, y, z) are the coordinates of the original point clouds in the depth sensor coordinate system, and T is a transformation matrix from the depth sensor coordinate system to the image acquisition device coordinate system.

### Claim 12 (depends on 11)

The device according to claim 11, wherein the transformation matrix T from the depth sensor coordinate system to the image acquisition device coordinate system satisfies:
T=T2T×T1,
wherein T1 is a transformation matrix from the depth sensor coordinate system to an unmanned aerial vehicle fuselage coordinate system, and T2 is a transformation matrix from the image acquisition device coordinate system to the unmanned aerial vehicle fuselage coordinate system.

### Claim 13 (depends on 9)

The device according to claim 9, wherein a coordinates (u, v) of the original point clouds in the plane where the target image is located satisfy:
d
×
(
u
v
)
=
k
×
(
x
′
y
′
z
′
)
⁢
and
k
=
(
f
x
0
c
x
0
f
y
c
y
0
0
1
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, d is the depth of the original point clouds, and k is an intrinsic matrix of the image acquisition device.

### Claim 14 (depends on 9)

The device according to claim 9, wherein the processor is further configured to
filter the original point clouds of which the coordinates fall into the minimum bounding box so as to remove the point clouds which do not belong to the target.

### Claim 15 (depends on 9)

The device according to claim 9, wherein the processor is further configured to
establish a three-dimensional map of the environment in front of the unmanned aerial vehicle according to the original point clouds of the environment in front of the unmanned aerial vehicle; and
take an average value of the three-dimensional coordinates of the point clouds corresponding to the target, wherein the average value is configured to represent the position of the target in the three-dimensional map.

### Claim 16 (depends on 9)

The device according to claim 9, wherein the depth sensor comprises at least one of a binocular camera, a structured light camera, a time of flight (TOF) camera, and a laser radar.

### Claim 17

A unmanned aerial vehicle, comprising
a fuselage;
an arm, connected to the fuselage;
a power unit, arranged on the arm and configured to provide power required by the flight of the unmanned aerial vehicle;
a depth sensor, arranged on the fuselage and configured to acquire original point clouds of an environment in front of the unmanned aerial vehicle;
an image acquisition device, arranged on the fuselage and configured to acquire a target image and acquire the position information of a target in the target image according to the target image; and
a visual chip, arranged on the fuselage, both the depth sensor and the image acquisition device being in communication connection with the visual chip, and
the visual chip being configured to:
acquire the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and
acquire the position information of the target in a three-dimensional space according to the point clouds corresponding to the target;
project the original point clouds to a plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located; and
determine the original point clouds of which the coordinates fall into a minimum bounding box in the original point clouds as the point clouds corresponding to the target.

### Claim 18 (depends on 17)

The unmanned aerial vehicle according to claim 17, wherein the visual chip is further configured to
transform the coordinates of the original point clouds in a depth sensor coordinate system into the coordinates of the original point clouds in an image acquisition device coordinate system; and
project the coordinates of the original point clouds in the image acquisition device coordinate system to the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located.

### Claim 19 (depends on 18)

The unmanned aerial vehicle according to claim 18, wherein the coordinates of the original point clouds in the image acquisition device coordinate system satisfy:
(
x
′
y
′
z
′
)
=
T
×
(
x
y
z
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, (x, y, z) are the coordinates of the original point clouds in the depth sensor coordinate system, and T is a transformation matrix from the depth sensor coordinate system to the image acquisition device coordinate system.

### Claim 20 (depends on 19)

The unmanned aerial vehicle according to claim 19, wherein the transformation matrix T from the depth sensor coordinate system to the image acquisition device coordinate system satisfies:
T=T2T×T1,
wherein T1 is the transformation matrix from the depth sensor coordinate system to a fuselage coordinate system, and T2 is the transformation matrix from the image acquisition device coordinate system to the fuselage coordinate system.

### Claim 21 (depends on 17)

The unmanned aerial vehicle according to claim 17, wherein a coordinates (u, v) of the original point clouds in the plane where the target image is located satisfy:
d
×
(
u
v
)
=
k
×
(
x
′
y
′
z
′
)
⁢
and
k
=
(
f
x
0
c
x
0
f
y
c
y
0
0
1
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, d is the depth of the original point clouds, and k is an intrinsic matrix of the image acquisition device.

### Claim 22 (depends on 17)

The unmanned aerial vehicle according to claim 17, wherein the visual chip is further configured to
filter the original point clouds of which the coordinates fall into the minimum bounding box so as to remove the point clouds which do not belong to the target.

### Claim 23 (depends on 17)

The unmanned aerial vehicle according to claim 17, wherein the visual chip is further configured to
establish a three-dimensional map of the environment in front of the unmanned aerial vehicle according to the original point clouds of the environment in front of the unmanned aerial vehicle; and
take an average value of the three-dimensional coordinates of the point clouds corresponding to the target, wherein the average value is configured to represent the position of the target in the three-dimensional map.

### Claim 24 (depends on 17)

The unmanned aerial vehicle according to claim 17, wherein the depth sensor comprises at least one of
a binocular camera, a structured light camera, a time of flight (TOF) camera, and a laser radar.