# US 2021/0333793 A1 · Indoor Monocular Navigation Method Based on Cross-Sensor Transfer Learning and System Thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333793 A1
- Document ID: US20210333793A1
- Status: Published application
- Kind: A1
- Application number: 16/931,653
- Filed: 2020-07-17
- Published: 2021-10-28
- Earliest claimed: 2020-04-24
- Assignees: Harbin Institute of Technology
- Applicants: Harbin Institute of Technology
- Inventors: Huijun GAO; Zhan LI; Xidi XUE; Weichao SUN; Xuebo YANG; Xinghu YU
- Web page: https://patentgrep.com/patent/US20210333793A1

## CPC

- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G01C 21/206: {specially adapted for indoor navigation}
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G05D 1/0221: {involving a learning process}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/096: Transfer learning
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present invention relates to an indoor monocular navigation method based on cross-sensor transfer learning and a system thereof. Determining an preliminary autonomous navigation model according to simulated laser radar data; acquiring actual single-line laser radar data and monocular camera data of the mobile robot simultaneously in an actual environment; determining the heading angle of the mobile robot according to the actual laser radar data; determining a laser radar monocular vision navigation model, according to the generated heading angle of the mobile robot and the monocular camera data at a the same moment and by using a Resnet18 network and a pre-trained YOLO v3 network; determining a heading angle of the mobile robot at the current moment, according to the acquired monocular camera data and by using the laser radar monocular vision navigation model; performing navigation of the mobile robot.

## Claims

### Claim 1

An indoor monocular navigation method based on cross-sensor transfer learning, comprising:
acquiring simulation single-line laser radar data of a mobile robot in a simulation model, where the simulation model is built in a Webots open source simulation environment; the mobile robot is a DJI Mavic Pro; the mobile robot is provided with an LMS 291 single-line laser radar; the simulation single-line laser radar data is a depth value;
determining an autonomous navigation model according to the simulation single-line laser radar data and by using a deterministic policy gradient deep reinforcement learning method, where the autonomous navigation model takes the laser radar data as an input and outputs a heading angle of the mobile robot;
acquiring actual single-line laser radar data and monocular camera data of the mobile robot at the same moment in an actual environment;
determining a heading angle of the mobile robot according to the actual single-line laser radar data and by using the autonomous navigation model;
determining a laser radar monocular vision navigation model, according to the heading angle of the mobile robot at the same moment and monocular camera data at a corresponding moment and by using a Resnet18 network and a pre-trained YOLO v3 network, where the laser radar monocular vision navigation model takes the monocular camera data as an input, and outputs a heading angle at a corresponding moment;
acquiring monocular camera data to be determined;
determining a heading angle of the mobile robot at a current moment, according to the monocular camera data to be determined and by using the laser radar monocular vision navigation model;
performing navigation of the mobile robot according to the heading angle at the current moment.

### Claim 2 (depends on 1)

The indoor monocular navigation method based on cross-sensor transfer learning according to claim 1, wherein the determining an autonomous navigation model according to the simulation single-line laser radar data and by using a deterministic policy gradient deep reinforcement learning method comprises:
using R=−k·dmin−1+Rp for determining a reward function, where R is the reward function; k is a constant, with a value of 0.5; dmin is a safety margin of the current mobile robot; Rp is a dynamic performance excitation item for the current mobile robot,
R
p
=
⁢
{
-
0.3
if
⁢
⁢
v
≈
0
+
0.4
if
⁢
⁢
v
≠
0
⁢
⁢
and
⁢
⁢
w
≠
0
+
0.1
else
⁢
,
where v is a linear velocity and w is an angular velocity.

### Claim 3 (depends on 1)

The indoor monocular navigation method based on cross-sensor transfer learning according to claim 1, wherein the determining a laser radar monocular vision navigation model, according to the heading angle of the mobile robot at the same moment and monocular camera data at a corresponding moment and by using a Resnet18 network and a pre-trained YOLO v3 network comprises:
labeling the monocular camera data according to the heading angle of the mobile robot at the same moment, to obtain a sample image.

### Claim 4 (depends on 3)

The indoor monocular navigation method based on cross-sensor transfer learning according to claim 3, wherein the determining a laser radar monocular vision navigation model, according to the heading angle of the mobile robot at the same moment and monocular camera data at a corresponding moment and by using a Resnet18 network and a pre-trained YOLO v3 network further comprises:
acquiring a Resnet18 network and a pre-trained YOLO v3 network;
inputting the sample image to the Resnet18 network and the pre-trained YOLO v3 network; merging pedestrian information vectors output by the pre-trained YOLO v3 network with a fully connected layer of the Resnet18 network, to obtain a laser radar monocular vision navigation model.

### Claim 5

An indoor monocular navigation system based on cross-sensor transfer learning, comprising:
a first acquisition module, configured to acquire simulation single-line laser radar data of a mobile robot in a simulation model, where the simulation model is built in a Webots open source simulation environment; the mobile robot is a DJI Mavic Pro; the mobile robot is provided with an LMS 291 single-line laser radar; the simulation single-line laser radar data is a depth value;
an autonomous navigation model determination module, configured to determine an autonomous navigation model according to the simulation single-line laser radar data and by using a deterministic policy gradient deep reinforcement learning method, where the autonomous navigation model takes the laser radar data as an input and outputs a heading angle of the mobile robot;
a second acquisition module, configured to acquire actual single-line laser radar data and monocular camera data of the mobile robot at the same moment in an actual environment;
a mobile robot heading angle determination module, configured to determine a heading angle of the mobile robot according to the actual single-line laser radar data and by using the autonomous navigation model.
a laser radar monocular vision navigation model determination module, configured to determine a laser radar monocular vision navigation model, according to the heading angle of the mobile robot at the same moment and monocular camera data at a corresponding moment and by using a Resnet18 network and a pre-trained YOLO v3 network, where the laser radar monocular vision navigation model takes the monocular camera data as an input, and outputs a heading angle at a corresponding moment;
a third acquisition module, configured to acquire monocular camera data to be determined
a mobile robot current moment heading angle determination module, configured to determine a heading angle of the mobile robot at a current moment, according to the monocular camera data to be determined and by using the laser radar monocular vision navigation model;
a navigation module, configured to perform navigation of the mobile robot according to the heading angle at the current moment.

### Claim 6 (depends on 5)

The indoor monocular navigation system based on cross-sensor transfer learning according to claim 5, wherein the autonomous navigation model determination module comprises:
a reward function determination unit, configured to determine a reward function by using R=−k·dmin−1+Rp, where R is the reward function; k is a constant, with a value of 0.5; dmin is a safety margin of the current mobile robot; Rp is a dynamic performance excitation item for the current mobile robot,
⁢
R
p
=
{
-
0.3
if
⁢
⁢
v
≈
0
+
0.4
if
⁢
⁢
v
≠
0
⁢
⁢
and
⁢
⁢
w
≠
0
+
0.1
else
⁢
,
where v is a linear velocity and w is an angular velocity.

### Claim 7 (depends on 5)

The indoor monocular navigation system based on cross-sensor transfer learning according to claim 5, wherein the laser radar monocular vision navigation model determination module comprises:
a sample image determination unit, configured to label the monocular camera data according to the heading angle of the mobile robot at the same moment, to obtain a sample image.

### Claim 8 (depends on 7)

The indoor monocular navigation system based on cross-sensor transfer learning according to claim 7, wherein the laser radar monocular vision navigation model determination module further comprises:
a first acquisition unit, configured to acquire a Resnet18 network and a pre-trained YOLO v3 network;
a laser radar monocular vision navigation model determination unit, configured to input the sample image to the Resnet18 network and the pre-trained YOLO v3 network, and merge a pedestrian information vector output by the pre-trained YOLO v3 network with a fully connected layer of the Resnet18 network to obtain a laser radar monocular vision navigation model.
