# US 2019/0057299 A1 · System for building a map and subsequent localization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0057299 A1
- Document ID: US20190057299A1
- Status: Published application
- Kind: A1
- Application number: 16/110,212
- Filed: 2018-08-23
- Published: 2019-02-21
- Earliest claimed: 2015-06-26
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Gershom Kutliroff
- Web page: https://patentgrep.com/patent/US20190057299A1

## CPC

- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G01C 21/3833: {characterised by the source of data}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24137: {Distances to cluster centroïds}
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06T 7/73: using feature-based methods
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06F 18/23213: with fixed number of clusters, e.g. K-means clustering
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

SLAM systems are provided that utilize an artificial neural network to both map environments and locate positions within the environments. In some example embodiments, a sensor arrangement is used to map an environment. The sensor arrangement acquires sensor data from the various sensors and associates the sensor data, or data derived from the sensor data, with spatial regions in the environment. The sensor data may include image data and inertial measurement data that effectively describes the visual appearance of a spatial region at a particular location and orientation. This diverse sensor data may be fused into camera poses. The map of the environment includes camera poses organized by spatial region within the environment. Further, in these examples, an artificial neural network is adapted to the features of the environment by a transfer learning process using image data associated with camera poses.

## Claims

### Claim 1

A simultaneous localization and mapping (SLAM) system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive sensor data descriptive of a plurality of spatial regions, the sensor data including image data;
train, using the image data, an artificial neural network to identify spatial regions of the plurality of spatial regions;
provide, to the artificial neural network, new sensor data descriptive of a spatial region of the plurality of spatial regions;
receive, from the artificial neural network, information identifying the spatial region of the plurality of spatial regions;
identify one or more keyframes associated with the spatial region; and
identify at least one camera pose using the one or more keyframes associated with the spatial region.

### Claim 2 (depends on 1)

The SLAM system of claim 1, wherein the sensor data includes one or more of three-dimensional image data, two-dimensional image data, and inertial measurement data.

### Claim 3 (depends on 1)

The SLAM system of claim 1, wherein the information identifying the spatial region of the plurality of spatial regions includes a list of spatial region identifiers and associated confidence levels.

### Claim 4 (depends on 1)

The SLAM system of claim 1, wherein the at least one processor is further configured to generate map data from the sensor data, the map data including one or more associations that associate the image data with one or more spatial regions of the plurality of spatial regions.

### Claim 5 (depends on 4)

The SLAM system of claim 4, wherein the image data includes the one or more keyframes, each of the one or more keyframes includes one or more keypoints, and the at least one processor is further configured to:
determine whether the new sensor data describes the one or more keypoints; and
identify the at least one camera pose by identifying one or more camera poses associated with the one or more keypoints in response to determining that the new sensor data describes the one or more keypoints.

### Claim 6 (depends on 4)

The SLAM system of claim 4, further comprising one or more sensors, wherein the at least one processor is further configured to:
receive a first message indicating that the one or more sensors are located in one spatial region of the plurality of spatial regions; and
generate an association of the one or more associations based on sensor data received after the first message.

### Claim 7 (depends on 6)

The SLAM system of claim 6, wherein the at least one processor is further configured to:
receive a second message requesting re-localization of the one or more sensors; and
provide the new sensor data to the artificial neural network in response to receiving the second message.

### Claim 8

A method of mapping and localizing within an environment using a computing device, the method comprising:
receiving sensor data descriptive of a plurality of spatial regions, the sensor data including image data;
training, using the image data, an artificial neural network to identify spatial regions of the plurality of spatial regions;
providing new sensor data descriptive of a spatial region of the plurality of spatial regions to the artificial neural network;
receiving, from the artificial neural network, information identifying the spatial region of the plurality of spatial regions;
identifying one or more keyframes associated with the spatial region; and
identifying at least one camera pose using the one or more keyframes associated with the spatial region.

### Claim 9 (depends on 8)

The method of claim 8, wherein receiving the sensor data includes receiving one or more of three-dimensional image data, two-dimensional image data, and inertial measurement data.

### Claim 10 (depends on 8)

The method of claim 8, wherein receiving the information identifying the spatial region of the plurality of spatial regions includes receiving a list of spatial region identifiers and associated confidence levels.

### Claim 11 (depends on 8)

The method of claim 8, further comprising generating map data from the sensor data, the map data associating the sensor data with one or more spatial regions of the plurality of spatial regions and associating keyframes comprising keypoints with the one or more spatial regions.

### Claim 12 (depends on 11)

The method of claim 11, wherein the image data includes the one or more keyframes, each of the one or more keyframes includes one or more keypoints, and the method further comprises:
determining whether the new sensor data describes the one or more keypoints; and
identifying the at least one camera pose by identifying one or more camera poses associated with the one or more keypoints in response to determining that the new sensor data describes the one or more keypoints.

### Claim 13 (depends on 11)

The method of claim 11, further comprising:
receiving a first message indicating that one or more sensors are located in one spatial region of the plurality of spatial regions; and
generating an association between sensor data received after the first message and the one spatial region.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
receiving a second message requesting re-localization of the one or more sensors; and
providing the new sensor data to the artificial neural network in response to receiving the second message.

### Claim 15

A non-transitory computer program product encoded with instructions that when executed by one or more processors cause a process for mapping and localizing within an environment using an artificial neural network to be carried out, the process comprising:
receiving sensor data descriptive of a plurality of spatial regions, the sensor data including image data;
training, using the image data, the artificial neural network to identify spatial regions of the plurality of spatial regions;
providing new sensor data descriptive of a spatial region of the plurality of spatial regions to the artificial neural network;
receiving information identifying the spatial region of the plurality of spatial regions from the artificial neural network;
identifying one or more keyframes associated with the spatial region; and
identifying at least one camera pose using the one or more keyframes associated with the spatial region.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein receiving the sensor data includes receiving one or more of three-dimensional image data, two-dimensional image data, and inertial measurement data.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein receiving the information identifying the spatial region of the plurality of spatial regions includes receiving a list of spatial region identifiers and associated confidence levels.

### Claim 18 (depends on 15)

The computer program product of claim 15, the process further comprising generating map data from the sensor data, the map data associating the sensor data with one or more spatial regions of the plurality of spatial regions.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the image data includes the one or more keyframes, each of the one or more keyframes includes one or more keypoints, and the process further comprises:
determining whether the new sensor data describes the one or more keypoints; and
identifying the at least one camera pose by identifying one or more camera poses associated with the one or more keypoints in response to determining that the new sensor data describes the one or more keypoints.

### Claim 20 (depends on 18)

The computer program product of claim 18, the process further comprising:
receiving a first message indicating that one or more sensors are located in one spatial region of the plurality of spatial regions; and
generating an association between sensor data received after the first message and the one spatial region.

### Claim 21 (depends on 20)

The computer program product of claim 20, the process further comprising:
receiving a second message requesting re-localization of the one or more sensors; and
providing the new sensor data to the artificial neural network in response to receiving the second message.
