# US 2024/0393996 A1 · Thumbnail for Disambiguation in a Visual Search

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0393996 A1
- Document ID: US20240393996A1
- Status: Published application
- Kind: A1
- Application number: 18/640,553
- Filed: 2024-04-19
- Published: 2024-11-28
- Earliest claimed: 2023-05-22
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Thomas J. Moore; Alesha Unpingco; Anshu K. Chimalamarri; Ashwin K. Vijay; Christopher D. Fu; Guilherme Klink; In Young Yang; Paul Ewers; Paulo R. Jansen dos Reis; Peter Burgner; Thomas G. Salter; Tigran Khachatryan
- Web page: https://patentgrep.com/patent/US20240393996A1

## CPC

- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A head-mounted device may be used to perform a visual search on a physical environment around the head-mounted device. A user may wish to visually search one out of multiple physical objects in the physical environment. To clearly show the user which physical object was the target of a visual search, the head-mounted device may present a thumbnail of the candidate physical object on a display. The thumbnail may be cropped and/or zoomed using an image from a camera on the head-mounted device. By displaying a thumbnail that is taken by a camera on the head-mounted device, the thumbnail will directly match the physical object in the user's physical environment, eliminating ambiguity regarding which physical object is the subject of the visual search.

## Claims

### Claim 1

An electronic device comprising:
one or more sensors;
one or more displays;
one or more processors; and
memory storing instructions configured to be executed by the one or more processors, the instructions for:
selecting a subset of a physical environment for a visual search;
obtaining, via a first subset of the one or more sensors, an image of the subset of the physical environment; and
presenting, using the one or more displays, a thumbnail that includes the image of the subset of the physical environment.

### Claim 2 (depends on 1)

The electronic device defined in claim 1, wherein the instructions further comprise instructions for:
presenting, using the one or more displays, visual search result information adjacent to the thumbnail.

### Claim 3 (depends on 2)

The electronic device defined in claim 2, wherein the instructions further comprise instructions for:
selecting an additional subset of the physical environment for an additional visual search;
obtaining, via the first subset of the one or more sensors, an additional image of the additional subset of the physical environment;
presenting, using the one or more displays and while presenting the thumbnail, an additional thumbnail that includes the additional image of the additional subset of the physical environment; and
presenting, using the one or more displays, additional visual search result information adjacent to the additional thumbnail.

### Claim 4 (depends on 1)

The electronic device defined in claim 1, wherein the instructions further comprise instructions for:
obtaining, via a second subset of the one or more sensors, a user request for a visual search.

### Claim 5 (depends on 1)

The electronic device defined in claim 1, wherein the instructions further comprise instructions for:
obtaining, via a third subset of the one or more sensors, sensor data, wherein selecting the subset of the physical environment for the visual search comprises selecting the subset of the physical environment for the visual search based on the sensor data.

### Claim 6 (depends on 5)

The electronic device defined in claim 5, wherein the third subset of the one or more sensors comprises an outward-facing camera that is configured to detect a hand gesture, a gaze detection sensor that is configured to determine a point of gaze, or an accelerometer that is configured to obtain head pose information.

### Claim 7 (depends on 1)

The electronic device defined in claim 1, wherein the one or more displays have an associated field-of-view and wherein the subset of the physical environment is outside of the field-of-view.

### Claim 8 (depends on 1)

The electronic device defined in claim 1, wherein the instructions further comprise instructions for:
analyzing a characteristic of the image of the subset of the physical environment;
in response to determining that the characteristic fails to satisfy a criterion, foregoing presenting the thumbnail; and
in response to determining that the characteristic fails to satisfy a criterion, presenting a photograph associated with the visual search of the subset of the physical environment, the photograph different than the obtained image of the subset of the physical environment.

### Claim 9 (depends on 1)

The electronic device defined in claim 1, wherein the instructions further comprise instructions for:
analyzing a characteristic of the image of the subset of the physical environment, wherein presenting the thumbnail comprises presenting the thumbnail in accordance with a determination that the characteristic satisfies a criterion.

### Claim 10

A method of operating an electronic device that comprises one or more sensors and one or more displays, the method comprising:
selecting a subset of a physical environment for a visual search;
obtaining, via a first subset of the one or more sensors, an image of the subset of the physical environment; and
presenting, using the one or more displays, a thumbnail that includes the image of the subset of the physical environment.

### Claim 11 (depends on 10)

The method defined in claim 10, further comprising:
presenting, using the one or more displays, visual search result information adjacent to the thumbnail.

### Claim 12 (depends on 11)

The method defined in claim 11, further comprising:
selecting an additional subset of the physical environment for an additional visual search;
obtaining, via the first subset of the one or more sensors, an additional image of the additional subset of the physical environment;
presenting, using the one or more displays and while presenting the thumbnail, an additional thumbnail that includes the additional image of the additional subset of the physical environment; and
presenting, using the one or more displays, additional visual search result information adjacent to the additional thumbnail.

### Claim 13 (depends on 10)

The method defined in claim 10, further comprising:
obtaining, via a second subset of the one or more sensors, a user request for a visual search.

### Claim 14 (depends on 10)

The method defined in claim 10, further comprising:
obtaining, via a third subset of the one or more sensors, sensor data, wherein selecting the subset of the physical environment for the visual search comprises selecting the subset of the physical environment for the visual search based on the sensor data.

### Claim 15 (depends on 14)

The method defined in claim 14, wherein the third subset of the one or more sensors comprises an outward-facing camera that is configured to detect a hand gesture, a gaze detection sensor that is configured to determine a point of gaze, or an accelerometer that is configured to obtain head pose information.

### Claim 16 (depends on 10)

The method defined in claim 10, wherein the one or more displays have an associated field-of-view and wherein the subset of the physical environment is outside of the field-of-view.

### Claim 17 (depends on 10)

The method defined in claim 10, further comprising:
analyzing a characteristic of the image of the subset of the physical environment;
in response to determining that the characteristic fails to satisfy a criterion, foregoing presenting the thumbnail; and
in response to determining that the characteristic fails to satisfy a criterion, presenting a photograph associated with the visual search of the subset of the physical environment, the photograph different than the obtained image of the subset of the physical environment.

### Claim 18 (depends on 10)

The method defined in claim 10, further comprising:
analyzing a characteristic of the image of the subset of the physical environment, wherein presenting the thumbnail comprises presenting the thumbnail in accordance with a determination that the characteristic satisfies a criterion.

### Claim 19

A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device that comprises one or more sensors and one or more displays, the one or more programs including instructions for:
selecting a subset of a physical environment for a visual search;
obtaining, via a first subset of the one or more sensors, an image of the subset of the physical environment; and
presenting, using the one or more displays, a thumbnail that includes the image of the subset of the physical environment.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium defined in claim 19, wherein the instructions further comprise instructions for:
presenting, using the one or more displays, visual search result information adjacent to the thumbnail.

### Claim 21 (depends on 20)

The non-transitory computer-readable storage medium defined in claim 20, wherein the instructions further comprise instructions for:
selecting an additional subset of the physical environment for an additional visual search;
obtaining, via the first subset of the one or more sensors, an additional image of the additional subset of the physical environment;
presenting, using the one or more displays and while presenting the thumbnail, an additional thumbnail that includes the additional image of the additional subset of the physical environment; and
presenting, using the one or more displays, additional visual search result information adjacent to the additional thumbnail.

### Claim 22 (depends on 19)

The non-transitory computer-readable storage medium defined in claim 19, wherein the instructions further comprise instructions for:
obtaining, via a second subset of the one or more sensors, a user request for a visual search.

### Claim 23 (depends on 19)

The non-transitory computer-readable storage medium defined in claim 19, wherein the instructions further comprise instructions for:
obtaining, via a third subset of the one or more sensors, sensor data, wherein selecting the subset of the physical environment for the visual search comprises selecting the subset of the physical environment for the visual search based on the sensor data.

### Claim 24 (depends on 23)

The non-transitory computer-readable storage medium defined in claim 23, wherein the third subset of the one or more sensors comprises an outward-facing camera that is configured to detect a hand gesture, a gaze detection sensor that is configured to determine a point of gaze, or an accelerometer that is configured to obtain head pose information.

### Claim 25 (depends on 19)

The non-transitory computer-readable storage medium defined in claim 19, wherein the one or more displays have an associated field-of-view and wherein the subset of the physical environment is outside of the field-of-view.

### Claim 26 (depends on 19)

The non-transitory computer-readable storage medium defined in claim 19, wherein the instructions further comprise instructions for:
analyzing a characteristic of the image of the subset of the physical environment;
in response to determining that the characteristic fails to satisfy a criterion, foregoing presenting the thumbnail; and
in response to determining that the characteristic fails to satisfy a criterion, presenting a photograph associated with the visual search of the subset of the physical environment, the photograph different than the obtained image of the subset of the physical environment.

### Claim 27 (depends on 19)

The non-transitory computer-readable storage medium defined in claim 19, wherein the instructions further comprise instructions for:
analyzing a characteristic of the image of the subset of the physical environment, wherein presenting the thumbnail comprises presenting the thumbnail in accordance with a determination that the characteristic satisfies a criterion.
