On-device image recognition

A method implemented in a mobile device, comprising:

• storing, on the mobile device, a first model that has been trained to approximate an output of a second model that classifies, according to a classification scheme, image data received as input, wherein the first model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
• sending, to a computer system that is external to the mobile device, a request for entity entries in a database managed by the computer system, the request including location data specifying a geographic location of the mobile device, wherein each entity entry includes:
• location data specifying a geographic location;
• data specifying an entity located at the geographic location;
• and
• one or more embeddings generated by the first model at the computer system, each of the one or more embeddings generated from one or more images received as input to the first model, and each of the images being an image that depicts the entity at the geographic location, and wherein each embedding are data generated by one of the intermediate layers;
• receiving, from the computer system, a proper subset of the entity entries in the database, the proper subset selected based on the geographic location of the mobile device, and storing the proper subset of the entity entries on the mobile device;
• capturing an image by the mobile device and providing the image as input to the first model to generate a captured image embedding;
• determining a set of matching embeddings in the proper subset of the entity entries, each matching embedding being an embedding that is determined to match the captured image embedding according to a match criterion;
• selecting, based on the set of matching embeddings, an entity entry in the proper subset of the entity entries as a matching entity entry;
• and
• providing, on a display device of the mobile device, display data describing the entity of the matching entity entry.

A system, comprising:

• a data processing apparatus;
• and
• a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
• accessing a first model that has been trained to generate an output that classifies, according to a classification scheme, image data received as input;
• using the first model to train a second model to approximate the output of the first model to classify, according to the classification scheme, image data received as input, wherein the second model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
• and
• after the second model is trained:
• accessing map data that specifies a plurality of geographic locations, wherein each geographic location in at least a subset of the geographic locations is associated with an entity located at the geographic location and one or more images that depict the entity at the geographic location;
• for each geographic location associated with an entity:
• for each image of the one or more images that depict the entity located at the geographic location, providing the image to the second model to generate an embedding for the image, wherein the embedding are data generated by one of the intermediate layers;
• associating each of the one or more embeddings generated by the second model with the geographic location;
• and
• storing, in a database, location data specifying the geographic location, the associated one or more embeddings, and data specifying the entity, as an associated entity entry for the entity.

A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

• accessing a first model that has been trained to generate an output that classifies, according to a classification scheme, image data received as input;
• using the first model to train a second model to approximate the output of the first model to classify, according to the classification scheme, image data received as input, wherein the second model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
• and
• after the second model is trained:
• accessing map data that specifies a plurality of geographic locations, wherein each geographic location in at least a subset of the geographic locations is associated with an entity located at the geographic location and one or more images that depict the entity at the geographic location;
• for each geographic location associated with an entity:
• for each image of the one or more images that depict the entity located at the geographic location, providing the image to the second model to generate an embedding for the image, wherein the embedding are data generated by one of the intermediate layers;
• associating each of the one or more embeddings generated by the second model with the geographic location;
• and
• storing, in a database, location data specifying the geographic location, the associated one or more embeddings, and data specifying the entity, as an associated entity entry for the entity.

A system, comprising:

• a data processing apparatus;
• and
• a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
• storing, on a mobile device, a first model that has been trained to approximate an output of a second model that classifies, according to a classification scheme, image data received as input, wherein the first model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
• sending, to a computer system that is external to the mobile device, a request for entity entries in a database managed by the computer system, the request including location data specifying a geographic location of the mobile device, wherein each entity entry includes:
• location data specifying a geographic location;
• data specifying an entity located at the geographic location;
• and
• one or more embeddings generated by the first model at the computer system, each of the one or more embeddings generated from one or more images received as input to the first model, and each of the images being an image that depicts the entity at the geographic location, and wherein each embedding are data generated by one of the intermediate layers;
• receiving, from the computer system, a proper subset of the entity entries in the database, the proper subset selected based on the geographic location of the mobile device, and storing the proper subset of the entity entries on the mobile device;
• capturing an image by the mobile device and providing the image as input to the first model to generate a captured image embedding;
• determining a set of matching embeddings in the proper subset of the entity entries, each matching embedding being an embedding that is determined to match the captured image embedding according to a match criterion;
• selecting, based on the set of matching embeddings, an entity entry in the proper subset of the entity entries as a matching entity entry;
• and
• providing, on a display device of the mobile device, display data describing the entity of the matching entity entry.

A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

• storing, on a mobile device, a first model that has been trained to approximate an output of a second model that classifies, according to a classification scheme, image data received as input, wherein the first model includes a plurality of connected layers including an output layer, an input layer, and a plurality of intermediate layers;
• sending, to a computer system that is external to the mobile device, a request for entity entries in a database managed by the computer system, the request including location data specifying a geographic location of the mobile device, wherein each entity entry includes:
• location data specifying a geographic location;
• data specifying an entity located at the geographic location;
• and
• one or more embeddings generated by the first model at the computer system, each of the one or more embeddings generated from one or more images received as input to the first model, and each of the images being an image that depicts the entity at the geographic location, and wherein each embedding are data generated by one of the intermediate layers;
• receiving, from the computer system, a proper subset of the entity entries in the database, the proper subset selected based on the geographic location of the mobile device, and storing the proper subset of the entity entries on the mobile device;
• capturing an image by the mobile device and providing the image as input to the first model to generate a captured image embedding;
• determining a set of matching embeddings in the proper subset of the entity entries, each matching embedding being an embedding that is determined to match the captured image embedding according to a match criterion;
• selecting, based on the set of matching embeddings, an entity entry in the proper subset of the entity entries as a matching entity entry;
• and
• providing, on a display device of the mobile device, display data describing the entity of the matching entity entry.