# US 10,769,428 B2 · On-device image recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,428 B2
- Document ID: US10769428B2
- Status: Granted
- Kind: B2
- Application number: 16/101,594
- Filed: 2018-08-13
- Published: 2020-02-13
- Granted: 2020-09-08
- Earliest claimed: 2018-08-13
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Abhanshu Sharma; Fedir Zubach; Thomas Binder; Lukas Mach; Sammy El Ghazzal; Matthew Sharifi
- Web page: https://patentgrep.com/patent/US10769428B2

## CPC

- G06F 16/587: using geographical or spatial information, e.g. location
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/00: Computing arrangements using knowledge-based models
- G06N 20/00: Machine learning
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/422: Technical drawings; Geographical maps
- G06F 16/387: using geographical or spatial information, e.g. location
- G06N 3/02: Neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a second model to approximate the output of a first model to classify, according to a classification scheme, image data received as input, and after the second model is trained accessing map data that specifies a plurality of geographic locations, and for each geographic location associated with an entity for each image of the one or more images that depict the entity located at the geographic location, providing the image to the second model to generate an embedding for the image, associating each of the one or more embeddings generated by the second model with the geographic location, and storing, in a database, location data specifying the geographic location, the associated one or more embeddings, and data specifying the entity, as an associated entity entry for the entity.

## Claims

### Claim 1

A method implemented in a data processing apparatus, comprising:
accessing a first model that has been trained to generate an output that classifies, according to a classification scheme, image data received as input;
using the first model to train a second model to approximate the output of the first model to classify, according to the classification scheme, image data received as input, wherein the second model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers; and
after the second model is trained:
accessing map data that specifies a plurality of geographic locations, wherein each geographic location in at least a subset of the geographic locations is associated with an entity located at the geographic location and one or more images that depict the entity at the geographic location;
for each geographic location associated with an entity:
for each image of the one or more images that depict the entity located at the geographic location, providing the image to the second model to generate an embedding for the image, wherein the embedding are data generated by one of the intermediate layers;
associating each of the one or more embeddings generated by the second model with the geographic location; and
storing, in a database, location data specifying the geographic location, the associated one or more embeddings, and data specifying the entity, as an associated entity entry for the entity.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving, from a mobile device, a request for entity entries in the database, the request including location data specifying a geographic location of the mobile device; and
selecting a proper subset of the entity entries in the database based on the geographic location of the mobile device, wherein each entity entry in the proper subset of the entity entries has location data that is determined to meet a proximity threshold that indicates the geographic location of the entity is determined to be proximate to the geographic location of the mobile device.

### Claim 3 (depends on 2)

The method of claim 2, wherein the proximity threshold is a value that specifies a geographic distance.

### Claim 4 (depends on 2)

The method of claim 2, wherein the proximity threshold is a value that specifies a pre-defined area that includes the geographic location of the mobile device.

### Claim 5 (depends on 1)

The method of claim 1, wherein the data specifying the entity includes an entity description describing the entity, and an entity name of the entity.

### Claim 6 (depends on 5)

The method of claim 5, wherein the entity is a business entity.

### Claim 7 (depends on 5)

The method of claim 5, wherein the entity is a landmark entity.

### Claim 8 (depends on 1)

The method of claim 1, wherein the second model is a convolutional neural network.

### Claim 9

A method implemented in a mobile device, comprising:
storing, on the mobile device, a first model that has been trained to approximate an output of a second model that classifies, according to a classification scheme, image data received as input, wherein the first model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
sending, to a computer system that is external to the mobile device, a request for entity entries in a database managed by the computer system, the request including location data specifying a geographic location of the mobile device, wherein each entity entry includes:
location data specifying a geographic location;
data specifying an entity located at the geographic location; and
one or more embeddings generated by the first model at the computer system, each of the one or more embeddings generated from one or more images received as input to the first model, and each of the images being an image that depicts the entity at the geographic location, and wherein each embedding are data generated by one of the intermediate layers;
receiving, from the computer system, a proper subset of the entity entries in the database, the proper subset selected based on the geographic location of the mobile device, and storing the proper subset of the entity entries on the mobile device;
capturing an image by the mobile device and providing the image as input to the first model to generate a captured image embedding;
determining a set of matching embeddings in the proper subset of the entity entries, each matching embedding being an embedding that is determined to match the captured image embedding according to a match criterion;
selecting, based on the set of matching embeddings, an entity entry in the proper subset of the entity entries as a matching entity entry; and
providing, on a display device of the mobile device, display data describing the entity of the matching entity entry.

### Claim 10 (depends on 9)

The method of claim 9, wherein selecting, based on the set of matching embeddings, the entity entry in the proper subset of the entity entries as the matching entity entry comprises:
for each entity entry having a matching embedding, determining a distance value based on the location data for the entity entry and the location data for the mobile device; and
selecting the entity entry have a smallest distance value relative to the distance values of the other entity entries having a matching embedding.

### Claim 11 (depends on 9)

The method of claim 9, wherein providing display data describing the entity of the matching entity entry comprises:
selecting a display location on the captured image; and
displaying the display data on the captured image at the display location.

### Claim 12 (depends on 11)

The method of claim 11, wherein selecting a display location on the captured image comprises:
determining, from the captured image embedding, an activation value that is a highest value relative to other activation values in the captured image embedding; and
selecting the display location that corresponds to the highest activation value in the captured image embedding.

### Claim 13 (depends on 12)

The method of claim 12, wherein the captured image is one of a plurality of image frames captured by the mobile device, and the determining the set of matching embeddings in the proper subset of the entity entries, the selecting the display location that corresponds to the activation value in the captured image embedding, and the displaying the display data on the captured image at the display location are done for the plurality of image frames.

### Claim 14 (depends on 11)

The method of any of claim 11, wherein:
the captured image is one of a plurality of image frames captured by the mobile device; and
the determining the set of matching embeddings in the proper subset of the entity entries, the selecting an entity entry in the proper subset of the entity entries as a matching entity entry, and the providing the display data describing the entity of the matching entity entry, are done for the plurality of image frames.

### Claim 15 (depends on 9)

The method of claim 9, further comprising:
receiving, from the computer system, a further proper subset of the entity entries in the database, the further proper subset selected based on an updated geographic location of the mobile device, and storing the further proper subset of entity entries on the mobile device;
capturing a further image by the mobile device and providing the further image as input to the first model to generate a further captured image embedding;
determining a set of matching embeddings in the further proper subset of the entity entries, each matching embedding being an embedding that is determined to match the further captured image embedding according to a match criterion;
selecting, based on the set of matching embeddings, an further entity entry in the further proper subset of the entity entries as a further matching entity entry; and
providing, on the display device of the mobile device, display data describing the entity of the further matching entity entry.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
deleting, from the mobile device, the proper subset of the entity entries when the mobile device is determined to be in the updated geographic location.

### Claim 17

A system, comprising:
a data processing apparatus; and
a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
accessing a first model that has been trained to generate an output that classifies, according to a classification scheme, image data received as input;
using the first model to train a second model to approximate the output of the first model to classify, according to the classification scheme, image data received as input, wherein the second model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers; and
after the second model is trained:
accessing map data that specifies a plurality of geographic locations, wherein each geographic location in at least a subset of the geographic locations is associated with an entity located at the geographic location and one or more images that depict the entity at the geographic location;
for each geographic location associated with an entity:
for each image of the one or more images that depict the entity located at the geographic location, providing the image to the second model to generate an embedding for the image, wherein the embedding are data generated by one of the intermediate layers;
associating each of the one or more embeddings generated by the second model with the geographic location; and
storing, in a database, location data specifying the geographic location, the associated one or more embeddings, and data specifying the entity, as an associated entity entry for the entity.

### Claim 18

A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:
accessing a first model that has been trained to generate an output that classifies, according to a classification scheme, image data received as input;
using the first model to train a second model to approximate the output of the first model to classify, according to the classification scheme, image data received as input, wherein the second model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers; and
after the second model is trained:
accessing map data that specifies a plurality of geographic locations, wherein each geographic location in at least a subset of the geographic locations is associated with an entity located at the geographic location and one or more images that depict the entity at the geographic location;
for each geographic location associated with an entity:
for each image of the one or more images that depict the entity located at the geographic location, providing the image to the second model to generate an embedding for the image, wherein the embedding are data generated by one of the intermediate layers;
associating each of the one or more embeddings generated by the second model with the geographic location; and
storing, in a database, location data specifying the geographic location, the associated one or more embeddings, and data specifying the entity, as an associated entity entry for the entity.

### Claim 19

A system, comprising:
a data processing apparatus; and
a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
storing, on a mobile device, a first model that has been trained to approximate an output of a second model that classifies, according to a classification scheme, image data received as input, wherein the first model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
sending, to a computer system that is external to the mobile device, a request for entity entries in a database managed by the computer system, the request including location data specifying a geographic location of the mobile device, wherein each entity entry includes:
location data specifying a geographic location;
data specifying an entity located at the geographic location; and
one or more embeddings generated by the first model at the computer system, each of the one or more embeddings generated from one or more images received as input to the first model, and each of the images being an image that depicts the entity at the geographic location, and wherein each embedding are data generated by one of the intermediate layers;
receiving, from the computer system, a proper subset of the entity entries in the database, the proper subset selected based on the geographic location of the mobile device, and storing the proper subset of the entity entries on the mobile device;
capturing an image by the mobile device and providing the image as input to the first model to generate a captured image embedding;
determining a set of matching embeddings in the proper subset of the entity entries, each matching embedding being an embedding that is determined to match the captured image embedding according to a match criterion;
selecting, based on the set of matching embeddings, an entity entry in the proper subset of the entity entries as a matching entity entry; and
providing, on a display device of the mobile device, display data describing the entity of the matching entity entry.

### Claim 20

A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:
storing, on a mobile device, a first model that has been trained to approximate an output of a second model that classifies, according to a classification scheme, image data received as input, wherein the first model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
sending, to a computer system that is external to the mobile device, a request for entity entries in a database managed by the computer system, the request including location data specifying a geographic location of the mobile device, wherein each entity entry includes:
location data specifying a geographic location;
data specifying an entity located at the geographic location; and
one or more embeddings generated by the first model at the computer system, each of the one or more embeddings generated from one or more images received as input to the first model, and each of the images being an image that depicts the entity at the geographic location, and wherein each embedding are data generated by one of the intermediate layers;
receiving, from the computer system, a proper subset of the entity entries in the database, the proper subset selected based on the geographic location of the mobile device, and storing the proper subset of the entity entries on the mobile device;
capturing an image by the mobile device and providing the image as input to the first model to generate a captured image embedding;
determining a set of matching embeddings in the proper subset of the entity entries, each matching embedding being an embedding that is determined to match the captured image embedding according to a match criterion;
selecting, based on the set of matching embeddings, an entity entry in the proper subset of the entity entries as a matching entity entry; and
providing, on a display device of the mobile device, display data describing the entity of the matching entity entry.