# US 2022/0044486 A1 · Visual search refinement for computer generated rendering environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044486 A1
- Document ID: US20220044486A1
- Status: Published application
- Kind: A1
- Application number: 17/508,010
- Filed: 2021-10-22
- Published: 2022-02-10
- Earliest claimed: 2019-08-01
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Eshan Verma; Daniel Ulbricht; Angela Blechschmidt; Mohammad Haris Baig; Chen-Yu Lee; Tanmay Batra
- Web page: https://patentgrep.com/patent/US20220044486A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

## Claims

### Claim 1

A method, comprising:
at a first device with a processor:
detecting a physical object based on obtained image content;
determining, based on a detected feature associated with the physical object, generic virtual content associated with the physical object;
providing the generic virtual content within a computer generated rendering (CGR) environment;
transmitting a portion of the image content or the feature to a second device;
receiving, from the second device, a response based on an identification of the physical object performed by the second device; and
updating the CGR environment by replacing the generic virtual content with object-specific content.

### Claim 2 (depends on 1)

The method of claim 1, wherein providing the generic virtual content within the CGR environment comprises replacing, enhancing, or supplementing the physical object with the generic virtual content.

### Claim 3 (depends on 1)

The method of claim 1, wherein the generic virtual content is determined from an asset store residing on the first device.

### Claim 4 (depends on 1)

The method of claim 1, wherein the object-specific content is based on information about the physical object.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining generic virtual content associated with the physical object is based on a coarse recognition of the physical object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the portion of the image excludes a second portion of the image content.

### Claim 7 (depends on 1)

The method of claim 1, wherein the feature associated with the physical object is a shape, a color, a size, a volume, or a marker.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining whether the image content satisfies privacy criteria; and
determining the portion of the image content based on the satisfaction of the privacy criteria.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
determining whether to transmit the portion of the image content based on privacy criteria.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
transmitting the detected feature to the second device.

### Claim 11 (depends on 1)

The method of claim 1, wherein the response associated with the physical object includes identification data to be used by the first device to identify the physical object in future image content.

### Claim 12 (depends on 1)

The method of claim 1, wherein the response associated with the physical object includes an assessment of a condition of the physical object.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first device or the second device is a head mounted device (HMD).

### Claim 14 (depends on 1)

The method of claim 1, wherein the first device or the second device is a mobile electronic device.

### Claim 15

A system comprising:
a first device with a processor; and
a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising:
detecting a physical object based on obtained image content;
determining, based on a detected feature associated with the physical object, generic virtual content associated with the physical object;
providing the generic virtual content within a computer generated rendering (CGR) environment;
transmitting a portion of the image content or the feature to a second device;
receiving, from the second device, a response based on an identification of the physical object performed by the second device; and
updating the CGR environment by replacing the generic virtual content with object-specific content.

### Claim 16 (depends on 15)

The system of claim 15, wherein providing the generic virtual content within the CGR environment comprises replacing, enhancing, or supplementing the physical object with the generic virtual content.

### Claim 17 (depends on 16)

The system of claim 16, wherein the generic virtual content is determined from an asset store residing on the first device.

### Claim 18 (depends on 16)

The system of claim 16, wherein the object-specific content is based on information about the physical object.

### Claim 19 (depends on 15)

The system of claim 15, wherein determining generic virtual content associated with the physical object is based on a coarse recognition of the physical object.

### Claim 20

A non-transitory computer-readable storage medium storing program instructions that are executable to perform operations comprising:
detecting, by a first device, a physical object based on obtained image content;
determining, based on a detected feature associated with the physical object, generic virtual content associated with the physical object;
providing the generic virtual content within a computer generated rendering (CGR) environment;
transmitting a portion of the image content or the feature to a second device;
receiving, from the second device, a response based on an identification of the physical object performed by the second device; and
updating the CGR environment by replacing the generic virtual content with object-specific content.
