# US 2019/0244057 A1 · Object recognition state indicators

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244057 A1
- Document ID: US20190244057A1
- Status: Published application
- Kind: A1
- Application number: 16/387,224
- Filed: 2019-04-17
- Published: 2019-08-08
- Earliest claimed: 2017-07-31
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Don Barnett; John DiMartile; Alison Lentz; Rachel Lara Been
- Web page: https://patentgrep.com/patent/US20190244057A1

## CPC

- G06V 20/20: in augmented reality scenes
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 10/235: {based on user input or interaction}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06F 16/532: Query formulation, e.g. graphical querying
- G06V 10/248: {by interactive preprocessing or interactive shape modelling, e.g. feature points assigned by a user}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems including computer programs encoded on a computer storage medium, for generating and displaying object recognition state indicators during object recognition processing of an image. In one aspect, a method includes performing object recognition on an image displayed in an application environment of an application on a user device using an object recognition model having multiple object recognition states including an identification state, where a candidate object in the image is positively identified, and one or more precursor states to the identification state, and where each of the precursor states has a different respective indicator for display within the image during the respective precursor state that visually emphasizes the candidate object and the identification state has a different respective indicator for display within the image during the identification state that visually emphasizes the positively identified object as being positively identified.

## Claims

### Claim 1

A computer-implemented method, comprising:
providing an image for display in an application environment of an application executing on a user device;
performing object recognition on the image by use of an object recognition model that has a plurality of object recognition states, where at least one object recognition state is an identification state that is achieved when a candidate object detected in the image is positively identified by the model as an object in the image, and at least one or more states are precursor states to the identification state, and each of the identification state and the one or more precursor states are indicated by a respective identification state indicator and respective precursor state indicators;
providing, to the application executing on the user device, data that causes each of the precursor state indicators to be displayed within the image in the application environment to visually emphasize the candidate object and to indicate the respective precursor state of the object recognition model and the identification state indicator to be displayed within the image to visually emphasize the candidate object as being positively identified and to indicate the identification state of the object recognition model;
in response to the candidate object being positively identified and indicated by a display of the identification state indictor over the candidate object, enabling selection of the identification state indicator by the user to trigger further search actions.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein enabling selection of the identification state indicator by the user to trigger further search actions comprises:
enabling a touch selection of the state indicator; and
in response to a touch selection of the state indicator, enabling a search query input by the user.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
receiving a search query input by the user in response to the enabling of the search query input;
combining an identification of the identified object with the search query input to form a combined user search query input;
providing the combined search query input to a search system; and
receiving and displaying search results responsive to the combined search query input from the search system.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein combining the identification of the identified object with the search query input to form a combined user search query input comprises disambiguating the search query input by the identification of the identified object.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the search query input is provided to an assistant executing on a user device.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein enabling selection of the identification state indicator by the user to trigger further search actions comprises:
enabling a touch selection of the state indicator; and
in response to a touch selection of the state indicator, providing information about the identified object within the application environment.

### Claim 7

A system comprising:
a data processing apparatus; and
a non-transitory computer readable medium storing instructions executable by the data processing apparatus and thus upon such execution cause the data processing apparatus to perform operations comprising:
providing an image for display in an application environment of an application executing on a user device;
performing object recognition on the image by use of an object recognition model that has a plurality of object recognition states, where at least one object recognition state is an identification state that is achieved when a candidate object detected in the image is positively identified by the model as an object in the image, and at least one or more states are precursor states to the identification state, and each of the identification state and the one or more precursor states are indicated by a respective identification state indicator and respective precursor state indicators;
providing, to the application executing on the user device, data that causes each of the precursor state indicators to be displayed within the image in the application environment to visually emphasize the candidate object and to indicate the respective precursor state of the object recognition model and the identification state indicator to be displayed within the image to visually emphasize the candidate object as being positively identified and to indicate the identification state of the object recognition model;
in response to the candidate object being positively identified and indicated by a display of the identification state indictor over the candidate object, enabling selection of the identification state indicator by the user to trigger further search actions.

### Claim 8 (depends on 7)

The system of claim 7, wherein enabling selection of the identification state indicator by the user to trigger further search actions comprises:
enabling a touch selection of the state indicator; and
in response to a touch selection of the state indicator, enabling a search query input by the user.

### Claim 9 (depends on 8)

The system of claim 8, further comprising:
receiving a search query input by the user in response to the enabling of the search query input;
combining an identification of the identified object with the search query input to form a combined user search query input;
providing the combined search query input to a search system; and
receiving and displaying search results responsive to the combined search query input from the search system.

### Claim 10 (depends on 9)

The system of claim 9, wherein combining the identification of the identified object with the search query input to form a combined user search query input comprises disambiguating the search query input by the identification of the identified object.

### Claim 11 (depends on 9)

The system of claim 9, wherein the search query input is provided to an assistant executing on a user device.

### Claim 12 (depends on 7)

The system of claim 7, wherein enabling selection of the identification state indicator by the user to trigger further search actions comprises:
enabling a touch selection of the state indicator; and
in response to a touch selection of the state indicator, providing information about the identified object within the application environment.

### Claim 13

A non-transitory computer storage medium storing instructions executable by a data processing apparatus and that upon such execution cause the one or more server computers to perform operations comprising:
providing an image for display in an application environment of an application executing on a user device;
performing object recognition on the image by use of an object recognition model that has a plurality of object recognition states, where at least one object recognition state is an identification state that is achieved when a candidate object detected in the image is positively identified by the model as an object in the image, and at least one or more states are precursor states to the identification state, and each of the identification state and the one or more precursor states are indicated by a respective identification state indicator and respective precursor state indicators;
providing, to the application executing on the user device, data that causes each of the precursor state indicators to be displayed within the image in the application environment to visually emphasize the candidate object and to indicate the respective precursor state of the object recognition model and the identification state indicator to be displayed within the image to visually emphasize the candidate object as being positively identified and to indicate the identification state of the object recognition model;
in response to the candidate object being positively identified and indicated by a display of the identification state indictor over the candidate object, enabling selection of the identification state indicator by the user to trigger further search actions.
