# US 2024/0005652 A1 · Object detection with instance detection and general scene understanding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005652 A1
- Document ID: US20240005652A1
- Status: Published application
- Kind: A1
- Application number: 18/214,214
- Filed: 2023-06-26
- Published: 2024-01-04
- Earliest claimed: 2019-09-09
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Aitor Aldoma Buchaca; Andreas N. Moeller; Michael Kuhn
- Web page: https://patentgrep.com/patent/US20240005652A1

## CPC

- G06V 10/82: using neural networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/20: in augmented reality scenes
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06V 20/64: Three-dimensional [3D] objects
- G06V 20/60: Type of objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Various implementations disclosed herein include devices, systems, and methods that determine a particular object instance in CGR environments. In some implementations, an object type of an object depicted in an image of a physical environment is identified. Then, a particular instance is determined based on the object type and the image. In some implementations, objects of the particular instance have a set of characteristics that differs from sets of characteristics associated with other instances of the object type. Then, the set of characteristics of the particular instance of the object depicted in the physical environment is obtained.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method comprising:
at an electronic device having a processor:
identifying an object type of an object depicted in an image of a physical environment;
determining a particular instance of the object based on the object type and the image, objects of the particular instance having a set of characteristics that differs from sets of characteristics associated with other instances of the object type;
obtaining the set of characteristics of the particular instance of the object depicted in the physical environment; and
combining the set of characteristics of the particular instance of the object with a CGR environment depicting the physical environment.

### Claim 22 (depends on 21)

The method of claim 21, wherein identifying an object type of an object depicted in an image includes detecting the object type of a plurality of object types of the object in the image using a first machine learning model.

### Claim 23 (depends on 21)

The method of claim 21, wherein determining a particular instance of the object based on the object type and the image comprises using features extracted from a portion of the image depicting the object to generate a representation of the particular instance of the object; and
determining the particular instance using the representation.

### Claim 24 (depends on 23)

The method of claim 23, wherein determining the particular instance using the representation comprises:
querying a first database of instances of the object type using the representation; and
receiving the particular instance from the first database.

### Claim 25 (depends on 24)

The method of claim 24, wherein obtaining the set of characteristics of the particular instance of the object depicted in the physical environment comprises accessing a database to receive information on materials, dimensions, physical properties, or visual properties of the particular instance of the object.

### Claim 26 (depends on 23)

The method of claim 23, wherein determining the particular instance using the representation comprises using a second machine learning model that inputs the object type and the representation of the particular instance of the object and outputs the particular instance of the object.

### Claim 27 (depends on 26)

The method of claim 26, wherein obtaining the set of characteristics of the particular instance of the object depicted in the physical environment comprises accessing a database to receive information on materials, dimensions, physical properties, or visual properties of the particular instance of the object.

### Claim 28 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform environment lighting in the CGR environment depicting the physical environment.

### Claim 29 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform scene understanding of the CGR environment depicting the physical environment.

### Claim 30 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform scene reconstruction in the CGR environment depicting the physical environment.

### Claim 31 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform material detection in the CGR environment depicting the physical environment.

### Claim 32 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform environment texturing in the CGR environment depicting the physical environment.

### Claim 33 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to generate reflections of virtual objects in the CGR environment or reflections of real objects of the CGR environment in the virtual objects of the CGR environment depicting the physical environment.

### Claim 34 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform physics simulations in the CGR environment depicting the physical environment.

### Claim 35 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to reconstruct object portions in the CGR environment that are not in the image depicting the physical environment.

### Claim 36 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to determine object or plane boundaries in the CGR environment depicting the physical environment.

### Claim 37 (depends on 21)

The method of claim 21, wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to effect removal of real objects from the CGR environment depicting the physical environment.

### Claim 38

A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
identifying an object type of an object depicted in an image of a physical environment;
determining a particular instance based on the object type and the image, objects of the particular instance having a set of characteristics that differs from sets of characteristics associated with other instances of the object type;
obtaining the set of characteristics of the particular instance of the object depicted in the physical environment; and
combining the set of characteristics of the particular instance of the object with a CGR environment depicting the physical environment.

### Claim 39

A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
identifying an object type of an object depicted in an image of a physical environment;
determining a particular instance based on the object type and the image, objects of the particular instance having a set of characteristics that differs from sets of characteristics associated with other instances of the object type;
obtaining the set of characteristics of the particular instance of the object depicted in the physical environment; and
combining the set of characteristics of the particular instance of the object with a CGR environment depicting the physical environment.

### Claim 40 (depends on 39)

The non-transitory computer-readable storage medium of claim 39, wherein determining a particular instance of the object based on the object type and the image comprises using features extracted from a portion of the image depicting the object to generate a representation of the particular instance of the object; and
determining the particular instance using the representation.
