# US 12,214,800 B2 · Method of processing data for autonomous vehicle, electronic device, storage medium and autonomous vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,214,800 B2
- Document ID: US12214800B2
- Status: Granted
- Kind: B2
- Application number: 17/975,689
- Filed: 2022-10-28
- Published: 2023-04-20
- Granted: 2025-02-04
- Earliest claimed: 2021-11-02
- Assignees: Apollo Intelligent Driving Technology (Beijing) Co., Ltd.
- Applicants: Apollo Intelligent Driving Technology (Beijing) Co., Ltd.
- Inventors: Wei Gao; Wendong Ding; Guowei Wan; Liang Peng
- Web page: https://patentgrep.com/patent/US12214800B2

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- G01C 21/28: with correlation of data from several navigational instruments
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60W 40/10: related to vehicle motion
- B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces}
- G01C 21/165: {combined with non-inertial navigation instruments}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/93: for anti-collision purposes
- G06T 7/50: Depth or shape recovery
- G06T 7/73: using feature-based methods
- B60W 2050/0005: {Processor details or data handling, e.g. memory registers or chip architecture}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2556/00: Input parameters relating to data
- G01S 5/16: using electromagnetic waves other than radio waves
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of processing data for an autonomous vehicle, an electronic device, a storage medium, and an autonomous vehicle are provided. The method includes: acquiring sensor data for the autonomous vehicle, wherein the sensor data includes inertial measurement data, LiDAR data, and visual image data; determining a first constraint factor for the inertial measurement data according to the inertial measurement data and the visual image data; determining a second constraint factor for the LiDAR data according to the inertial measurement data and the LiDAR data; determining a third constraint factor for the visual image data according to the inertial measurement data, the visual image data and the LiDAR data; and processing the sensor data based on the first constraint factor, the second constraint factor and the third constraint factor, so as to obtain positioning data for positioning the autonomous vehicle.

## Claims

### Claim 1

A method of processing data for an autonomous vehicle, comprising:
acquiring sensor data for the autonomous vehicle, wherein the sensor data comprises inertial measurement data, LiDAR data, and visual image data;
determining a first constraint factor for the inertial measurement data according to the inertial measurement data and the visual image data;
determining a second constraint factor for the LiDAR data according to the inertial measurement data and the LiDAR data;
determining a third constraint factor for the visual image data according to the inertial measurement data, the visual image data, and the LiDAR data;
processing the sensor data based on the first constraint factor, the second constraint factor, and the third constraint factor, so as to obtain positioning data; and
positioning the autonomous vehicle by using the positioning data, wherein
the determining a third constraint factor for the visual image data according to the inertial measurement data, the visual image data, and the LiDAR data includes:
determining a depth information of the visual image data according to the visual image data and the LiDAR data;
determining a mapping relationship between the visual image data and the inertial measurement data according to the visual image data and the inertial measurement data; and
determining the third constraint factor according to the depth information and the mapping relationship, and
the determining a depth information of the visual image data according to the visual image data and the LiDAR data includes:
determining an initial inertial measurement relative pose at a visual image acquisition time instant of the visual image data relative to a radar data acquisition time instant of adjacent radar data;
determining a first projection position of the LiDAR data to a visual image data plane according to the visual image acquisition time instant, the radar data acquisition time instant, and the initial inertial measurement relative pose;
determining an image projection time instant and an updated inertial measurement relative pose according to the first projection position; and
determining the depth information of the visual image data according to the updated inertial measurement relative pose.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the LiDAR data comprises a plurality of frames of laser point cloud data; and the determining a second constraint factor for the LiDAR data according to the inertial measurement data and the LiDAR data comprises:
determining an initial relative pose between a first frame of laser point cloud data and a second frame of laser point cloud data among the plurality of frames of laser point cloud data according to the inertial measurement data, the first frame of laser point cloud data and the second frame of laser point cloud data;
determining first projection data of the first frame of laser point cloud data and second projection data of the second frame of laser point cloud data, respectively;
adjusting the initial relative pose according to the first projection data and the second projection data, so as to obtain a relative pose between point cloud frames; and
determining the second constraint factor according to the relative pose between point cloud frames.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the first projection data comprises N voxels, the second projection data comprises N voxels, and N is an integer greater than 1; the adjusting the initial relative pose according to the first projection data and the second projection data, so as to obtain a relative pose between point cloud frames comprises:
performing N iterations according to the initial relative pose and each of the N voxels, so as to obtain an Nth relative pose as the relative pose between point cloud frames.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the first projection data comprises a first projection image and a first normal vector, and the second projection data comprises a second projection image and a second normal vector; the adjusting the initial relative pose according to the first projection data and the second projection data, so as to obtain a relative pose between point cloud frames comprises:
in an ith iteration of the N iterations:
determining a distance threshold and an angle threshold according to i, where i is greater than or equal to 1 and less than or equal to N;
determining, in response to a determination that a distance between an ith first pixel in the first projection image and an ith second pixel in the second projection image is less than or equal to the distance threshold and that an angle between the first normal vector corresponding to an ith pixel and the second normal vector corresponding to the ith pixel is less than or equal to the angle threshold, an ith relative pose by using an (i−1)th relative pose obtained from an (i−1)th iteration, the ith first pixel, the ith second pixel, the first normal vector corresponding to the ith pixel, and the second normal vector corresponding to the ith pixel.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the determining an image projection time instant and an updated inertial measurement relative pose according to the first projection position comprises:
determining a first inertial measurement relative pose at the visual image acquisition time instant relative to the image projection time instant;
determining a second inertial measurement relative pose at the image projection time instant relative to the laser data acquisition time instant; and
performing an interpolation on the initial inertial measurement relative pose by using the first inertial measurement relative pose and the second inertial measurement relative pose, so as to obtain the updated inertial measurement relative pose.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the processing the sensor data based on the first constraint factor, the second constraint factor and the third constraint factor, so as to obtain positioning data comprises: determining X as the positioning data, wherein the X satisfies:
min
X
{
∑

r
imu
(
z
^
b
k
+
1
b
k
,
X
)

2
+
∑

r
v
⁢
i
⁢
s
⁢
i
⁢
o
⁢
n
(
z
ˆ
l
c
j
,
X
)

2
+
∑

r
icp
(
z
ˆ
s
e
,
X
)

2
}
where rimu
(
z
ˆ
b
k
+
1
b
k
,
X
)
represents the first constraint factor, rvision({circumflex over (z)}lcj,X) represents the third constraint factor, ricp({circumflex over (z)}se,X) represents the second constraint factor;
z
ˆ
b
k
+
1
b
k
represents the inertial measurement data, {circumflex over (z)}lcj represents the visual image data, and {circumflex over (z)}se represents the LIDAR data.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the inertial measurement data is acquired by an inertial sensor; and the method further comprises:
initializing a velocity of the inertial sensor and an acceleration of the inertial sensor according to the LiDAR data and the first constraint factor.

### Claim 8 (depends on 1)

An electronic device, comprising:
at least one processor; and
a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to implement the method of claim 1.

### Claim 9 (depends on 8)

The electronic device according to claim 8, wherein the LiDAR data comprises a plurality of frames of laser point cloud data; and the at least one processor is further configured for:
determining an initial relative pose between a first frame of laser point cloud data and a second frame of laser point cloud data among the plurality of frames of laser point cloud data according to the inertial measurement data, the first frame of laser point cloud data and the second frame of laser point cloud data;
determining first projection data of the first frame of laser point cloud data and second projection data of the second frame of laser point cloud data, respectively;
adjusting the initial relative pose according to the first projection data and the second projection data, so as to obtain a relative pose between point cloud frames; and
determining the second constraint factor according to the relative pose between point cloud frames.

### Claim 10 (depends on 9)

The electronic device according to claim 9, wherein the first projection data comprises N voxels, the second projection data comprises N voxels, and N is an integer greater than 1; and the at least one processor is further configured for:
performing N iterations according to the initial relative pose and each of the N voxels, so as to obtain an Nth relative pose as the relative pose between point cloud frames.

### Claim 11 (depends on 10)

The electronic device according to claim 10, wherein the first projection data comprises a first projection image and a first normal vector, and the second projection data comprises a second projection image and a second normal vector; the at least one processor is further configured for:
in an ith iteration of the N iterations:
determining a distance threshold and an angle threshold according to i, where i is greater than or equal to 1 and less than or equal to N;
determining, in response to a determination that a distance between an ith first pixel in the first projection image and an ith second pixel in the second projection image is less than or equal to the distance threshold and that an angle between the first normal vector corresponding to an ith pixel and the second normal vector corresponding to the ith pixel is less than or equal to the angle threshold, an ith relative pose by using an (i−1)th relative pose obtained from an (i−1)th iteration, the ith first pixel, the ith second pixel, the first normal vector corresponding to the ith pixel, and the second normal vector corresponding to the ith pixel.

### Claim 12 (depends on 8)

The electronic device according to claim 8, wherein the at least one processor is further configured for:
determining a first inertial measurement relative pose at the visual image acquisition time instant relative to the image projection time instant;
determining a second inertial measurement relative pose at the image projection time instant relative to the laser data acquisition time instant; and
performing an interpolation on the initial inertial measurement relative pose by using the first inertial measurement relative pose and the second inertial measurement relative pose, so as to obtain the updated inertial measurement relative pose.

### Claim 13 (depends on 8)

The electronic device according to claim 8, wherein the at least one processor is further configured for: determining X as the positioning data, wherein the X satisfies:
min
X
{
∑

r
imu
(
z
^
b
k
+
1
b
k
,
X
)

2
+
∑

r
v
⁢
i
⁢
s
⁢
i
⁢
o
⁢
n
(
z
ˆ
l
c
j
,
X
)

2
+
∑

r
icp
(
z
ˆ
s
e
,
X
)

2
}
where rimu
(
z
ˆ
b
k
+
1
b
k
,
X
)
represents the first constraint factor, rvision({circumflex over (z)}lcj,X) represents the third constraint factor, ricp({circumflex over (z)}se,X) represents the second constraint factor;
z
ˆ
b
k
+
1
b
k
represents the inertial measurement data, {circumflex over (z)}lcj represents the visual image data, and {circumflex over (z)}se represents the LiDAR data.

### Claim 14 (depends on 8)

The electronic device according to claim 8, wherein the inertial measurement data is acquired by an inertial sensor; and the at least one processor is further configured for:
initializing a velocity of the inertial sensor and an acceleration of the inertial sensor according to the LiDAR data and the first constraint factor.

### Claim 15 (depends on 1)

A non-transitory computer-readable storage medium having computer instructions therein, wherein the computer instructions are configured to cause a computer to implement the method of claim 1.

### Claim 16 (depends on 8)

An autonomous vehicle, comprising the electronic device according to claim 8.