# US 10,767,996 B2 · System and methods for reducing the map search space requirements in a vision-inertial navigation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,767,996 B2
- Document ID: US10767996B2
- Status: Granted
- Kind: B2
- Application number: 15/974,629
- Filed: 2018-05-08
- Published: 2019-11-14
- Granted: 2020-09-08
- Earliest claimed: 2018-05-08
- Assignees: Honeywell International Inc.
- Applicants: Honeywell International Inc.
- Inventors: Benjamin Mohr; Vijay Venkataraman
- Web page: https://patentgrep.com/patent/US10767996B2

## CPC

- G01C 21/005: {with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence)}
- G01C 21/1656: {with passive imaging devices, e.g. cameras}
- G01C 21/387: {Organisation of map data, e.g. version management or database structures}
- G06F 18/24: Classification techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/11: Region-based segmentation
- G06T 17/05: Geographic models
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system and methods for reducing map search area requirements in a navigation system are disclosed. The system includes a vehicle, an imaging device onboard the vehicle configured to generate an image scan, receive at least one image responsive to the image scan, and a processing device configured to receive and store the at least one image. The system further includes a navigation system onboard the vehicle configured to store an image of a map, and a learning network associated with the navigation system and configured to divide the image of the map into a plurality of map subsections, recognize each one of a plurality of images of different landmarks on the map, generate a set of classifications for each map subsection, and associate each classification of the set of classifications with at least one landmark of the different landmarks on the map.

## Claims

### Claim 1

A method for reducing map search area requirements in a navigation system, comprising:
training a learning algorithm to recognize each one of a plurality of images of different landmarks on a map;
dividing an image of the map into a plurality of subsections;
deriving a set of classifications for each subsection of the plurality of subsections, wherein each classification of the set of classifications is associated with at least one landmark of the different landmarks on the map;
receiving at least one image from an imaging device onboard a vehicle in flight;
deriving a classification for the at least one image from the imaging device onboard the vehicle in flight; and
comparing the classification for the at least one image with the set of classifications for each subsection of the plurality of subsections of the map, and limiting the comparing to the subsections of the map that have a substantially similar classification to the classification of the at least one image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the deriving the set of classifications comprises deriving a set of vector classification probabilities.

### Claim 3 (depends on 1)

The method of claim 1, wherein the training comprises training a Deep Learning artificial neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein the method comprises reducing the map search area requirements in a Visual-Inertial Navigation System (VINS).

### Claim 5 (depends on 1)

The method of claim 1, wherein the receiving the at least one image from the imaging device comprises receiving the at least image from a camera, an imaging device operating in a visible frequency band, an imaging device operating in an infra-red frequency band, an imaging device operating in an ultra-violet frequency band, or an imaging device operating in a millimeter frequency band.

### Claim 6 (depends on 1)

The method of claim 1, wherein the receiving at least one image from the imaging device onboard the vehicle in flight comprises receiving the at least one image from the imaging device onboard an aircraft, an unmanned aerial vehicle (UAV), a spacecraft, a missile, a guided bomb, or a large caliber projectile.

### Claim 7 (depends on 1)

The method of claim 1, wherein the dividing comprises a neural network component in a VINS dividing a base map into a plurality of non-overlapping subsections.

### Claim 8 (depends on 1)

The method of claim 1, wherein the comparing comprises comparing a first combination of classification probabilities with a second combination of classification probabilities.

### Claim 9 (depends on 1)

The method of claim 1, wherein the comparing comprises identifying a plurality of classes in an image, and comparing the plurality of classes in the image with the set of classifications for each subsection of the plurality of subsections of the map.

### Claim 10 (depends on 9)

The method of claim 9, further comprising limiting the identifying to at least one observed classification that is unique or rare, or to a sequence of observed classifications that are unique or rare.

### Claim 11

A system, comprising:
a vehicle;
an imaging device onboard the vehicle, wherein the imaging device is configured to generate an image scan, and receive at least one image responsive to the image scan;
a processing device, wherein the processing device is configured to receive and store the at least one image;
a navigation system onboard the vehicle, wherein the navigation system is configured to store an image of a map; and
a learning network associated with the navigation system, wherein the learning network is configured to divide the image of the map into a plurality of map subsections, recognize each one of a plurality of images of different landmarks on the map, generate a set of classifications for each map subsection, and associate each classification of the set of classifications with at least one landmark of the different landmarks on the map.

### Claim 12 (depends on 11)

The system of claim 11, wherein the learning network is further configured to receive the at least one image responsive to the image scan, generate a classification for the at least one image, compare the classification for the at least one image with the set of classifications for each map subsection, and limit the comparison to the map subsections that have a substantially similar classification to that of the at least one image.

### Claim 13 (depends on 11)

The system of claim 11, wherein the learning network is a Deep neural network.

### Claim 14 (depends on 11)

The system of claim 11, wherein the system is a VINS.

### Claim 15 (depends on 11)

The system of claim 11, wherein the vehicle is at least one of an aircraft, a UAV, a spacecraft, a missile, a guided bomb, or a large caliber projectile.

### Claim 16 (depends on 11)

The system of claim 11, wherein the imaging device is at least one of a camera, a device configured to receive images in a visible frequency band, a device configured to receive images in an infra-red frequency band, a device configured to receive images in an ultra-violet frequency band, or a device configured to receive images in a millimeter frequency band.

### Claim 17

A method, comprising:
providing aerial map imagery of an area associated with a prospective mission for a vehicle in flight;
dividing the aerial map imagery into a plurality of map subsections;
recognizing a plurality of landmarks in the plurality of map subsections;
generating a list of the recognized landmarks and their respective locations in the plurality of map subsections;
providing real-time aerial imagery during the prospective mission;
recognizing a plurality of landmarks in the real-time imagery and generating a second list of the recognized landmarks in the real-time imagery;
comparing the second list of the recognized plurality of landmarks in the real-time imagery with the list of the recognized landmarks in the plurality of map subsections; and
matching the recognized landmarks in the real-time imagery with the recognized landmarks in the map subsections.

### Claim 18 (depends on 17)

The method of claim 17, wherein the matching comprises generating a set of vector classification probabilities for the recognized landmarks in the real-time imagery, generating a second set of vector classification probabilities for the recognized landmarks in the map subsections, and matching the set of vector classification probabilities with the second set of classification probabilities.

### Claim 19 (depends on 17)

The method of claim 17, wherein the providing the aerial map imagery of the area associated with the prospective mission for the vehicle in flight comprises a neural network receiving the aerial map imagery.

### Claim 20 (depends on 17)

The method of claim 17, further comprising utilizing position information for landmarks recognized in the aerial map imagery and the current camera image, and thereby reducing uncertainties associated with the positions of the recognized landmarks.