# US 2023/0410498 A1 · Cycling performing image classification based on user familiarity

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410498 A1
- Document ID: US20230410498A1
- Status: Published application
- Kind: A1
- Application number: 18/248,148
- Filed: 2020-10-13
- Published: 2023-12-21
- Earliest claimed: 2020-10-13
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Brett Barros; Megan Fazio
- Web page: https://patentgrep.com/patent/US20230410498A1

## CPC

- G06V 10/96: Management of image or video recognition tasks
- G06F 9/453: {Help systems}
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06V 10/17: {using hand-held instruments}
- G06V 10/764: using classification, e.g. of video objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computing system can perform image classification within a displayed image, while performing the image classification, determine that a first object of an object class is present within the displayed image, present a prompt within the displayed image, receive, in response to the prompt, a selection of the first object from a user, in response to receiving the selection of the first object, perform a function on the first object, based on the selection of the first object in response to the prompt, determine that the user is familiar with the function, based on determining that the user is familiar with the function, terminate performing image classification within the displayed image, and in response to the user selecting a second object of the object class within the displayed image, perform the function on the second object.

## Claims

### Claim 1

A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to:
perform image classification within a displayed image;
while performing the image classification, determine that a first object of an object class is present within the displayed image;
present a prompt within the displayed image;
receive, in response to the prompt, a selection of the first object from a user;
in response to receiving the selection of the first object, perform a function on the first object;
based on the selection of the first object in response to the prompt, determine that the user is familiar with the function;
based on determining that the user is familiar with the function, terminate performing image classification within the displayed image; and
in response to the user selecting a second object of the object class within the displayed image, perform the function on the second object.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the performing the function on the first object comprises performing a local image recognition function on the displayed image.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the performing the function on the first object comprises:
sending the displayed image to a server; and
receiving a transformed object from the server.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the object class includes text.

### Claim 5 (depends on 4)

The non-transitory computer-readable storage medium of claim 4, wherein the function includes performing optical character recognition on the first object.

### Claim 6 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the object class includes a barcode.

### Claim 7 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein the function includes decoding the first object.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the object class includes humans.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the function includes determining a name associated with the first object.

### Claim 10 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions are further configured to cause the computing system to determine the object class based on a location of the computing system.

### Claim 11 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions are further configured to cause the computing system to determine the object class based on images previously viewed by the user.

### Claim 12 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein with the instructions are further configured to cause the computing system to determine whether an object is of an object class that the computing system can perform a function on.

### Claim 13 (depends on 1)

The non-transitory storage medium of claim 1, wherein the determination of familiarity by a familiarity determiner depends on a number of selections of an object.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the familiarity determiner controls the termination of the classification if a predetermined first threshold for a familiarity value is reached and controls the resumption of the classification if a predetermined second threshold for the familiarity value is reached.

### Claim 15 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein a context processor processes context information that the computing system can combine with a familiarity value to determine whether to perform or not perform image classification and/or present a prompt, the context information being in particular a charge level of a battery included in the computing system, a time of day, an audio signal received by a microphone included in the computing system, a location of the computing system, or an interest of a user in images of a particular object class as shown by the user viewing images of the object class and/or storing images of the object class.

### Claim 16 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein:
the determination that the user is familiar with the function comprises determining that the user is familiar with the function with respect to the object class; and
the terminating performing image classification within the displayed image comprises terminating performing image classification within the displayed image with respect to the object class.

### Claim 17 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions are configured to cause the computing system to terminate performing image classification within the displayed image based on determining that the user is familiar with the function and a time of day.

### Claim 18 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions are configured to cause the computing system to terminate performing image classification within the displayed image based on determining that the user is familiar with the function and a charge level of a battery included in the computing system.

### Claim 19 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions are configured to cause the computing system to terminate performing image classification within the displayed image based on determining that the user is familiar with the function and audio signals received by a microphone included in the computing system.

### Claim 20 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein:
the instructions are further configured to cause the computing system to perform image detection on the first object after determining that the first object of the object class is present within the displayed image; and
the presenting the prompt within the displayed image comprises presenting a prompt in a location within the displayed image based on a location where the first object was detected within the displayed image.

### Claim 21 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions are further configured to cause the computing system to:
display multiple additional images;
based on the user not selecting portions of the displayed multiple additional images, determine that the user is no longer familiar with the function; and
based on determining that the user is no longer familiar with the function:
perform image classification within a subsequently displayed image;
while performing the image classification within the subsequently displayed image, determine that a subsequent object of the object class is present within the subsequently displayed image; and
present a subsequent prompt within the subsequently displayed image.

### Claim 22 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein:
the object class comprises a first object class;
the prompt comprises a first prompt;
the function comprises a first function; and
the instructions are further configured to cause the computing system to:
perform image classification within a subsequently displayed image;
while performing the image classification within the subsequently displayed image, detect a second object of a second object class within the subsequently displayed image, the second object class being different than the first object class;
present a second prompt within the subsequently displayed image;
receive a selection of the second object from the user; and
in response to receiving the selection of the second object, perform a second function on the second object, the second function being different than the first function.

### Claim 23

A computing system comprising:
at least one processor; and
a non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause the computing system to:
perform image classification within a displayed image;
while performing the image classification, determine that a first object of an object class is present within the displayed image;
present a prompt within the displayed image;
receive, in response to the prompt, a selection of the first object from a user;
in response to receiving the selection of the first object, perform a function on the first object;
based on the selection of the first object in response to the prompt, determining that the user is familiar with the function;
based on determining that the user is familiar with the function, terminate performing image classification within the displayed image; and
in response to the user selecting a second object of the object class within the displayed image, perform the function on the second object.

### Claim 24

A method comprising:
performing, by a computing system, image classification within a displayed image;
while performing the image classification, determining that a first object of an object class is present within the displayed image;
presenting a prompt within the displayed image;
receiving a selection of the first object from a user;
in response to receiving the selection of the first object, performing a function on the first object;
based on the selection of the first object in response to the prompt, determining that the user is familiar with the function;
based on determining that the user is familiar with the function, terminating performing image classification within the displayed image; and
in response to the user selecting a second object of the object class within the displayed image, performing the function on the second object.

### Claim 25

A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to:
perform image classification within a first displayed image;
while performing the image classification, determine that a first object of an object class is present within the first displayed image;
present a prompt within the first displayed image;
receive a selection of the first object from a user;
in response to receiving the selection of the first object, perform a function on the first object;
based on the selection of the first object in response to the prompt, determine that the user is familiar with the function;
based on determining that the user is familiar with the function, present a second displayed image without performing image classification within the second displayed image; and
in response to the user selecting a second object of the object class within the second displayed image, perform the function on the second object.
