# US 2023/0410540 A1 · Systems and methods for mapping an environment and locating objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410540 A1
- Document ID: US20230410540A1
- Status: Published application
- Kind: A1
- Application number: 17/951,826
- Filed: 2022-09-23
- Published: 2023-12-21
- Earliest claimed: 2022-06-17
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Afshin DEHGHAN; Angela BLECHSCHMIDT; Yuanzheng GONG; Feng TANG; Yang YANG; Zhihao ZHU; Monica Laura ZUENDORF
- Web page: https://patentgrep.com/patent/US20230410540A1

## CPC

- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 20/20: in augmented reality scenes
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and processes for locating objects and environmental mapping are provided. For example, a device may receive an input including a reference to a first object, wherein the input includes a request for a location of the first object, and the first object is located in a physical environment. The device may determine a location of the first object, and in response to receiving the input, if first criteria are met, provide a description of the location, wherein a criterion of the one or more first criteria is met when the location is available, and the description includes a relationship between the first object and a reference within the physical environment. If second criteria are met, the device may forgo providing the description of the location, wherein a criterion of the one or more second criteria is met when the location is not available.

## Claims

### Claim 1

An electronic device, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving an input including a reference to a first object, wherein
the input includes a request for a location of the first object, and
the first object is located in a physical environment;
determining a location of the first object; and
in response to receiving the input:
in accordance with a determination that one or more first criteria are met, providing a description of the location of the first object, wherein
a criterion of the one or more first criteria is met when the location of the first object is available, and
the description includes a relationship between the first object and a reference within the physical environment; and
in accordance with a determination that one or more second criteria are met, forgoing providing the description of the location of the first object, wherein a criterion of the one or more second criteria is met when the location of the first object is not available.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the first object corresponds to a second electronic device different from the electronic device.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein the first object corresponds to the electronic device.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the description includes a reference to a second object different from the first object.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein
the reference within the physical environment corresponds to an identified area within the physical environment, and
the relationship between the first object and the reference within the physical environment includes the first object within the identified area.

### Claim 6 (depends on 1)

The electronic device of claim 1, the one or more programs including instructions for:
in response to receiving the input including the reference to the first object, requesting location information for the first object;
receiving location information from the first object; and
determining the location of the first object based on the location information.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein determining the location of the first object comprises:
obtaining a representation of the physical environment;
comparing the location information with location information corresponding to the representation; and
determining, based on the comparison, the location of the first object.

### Claim 8 (depends on 7)

The electronic device of claim 7, wherein the location information corresponding to the representation includes a plurality of object locations, each object location of the plurality of object locations corresponding to a respective physical object in the physical environment.

### Claim 9 (depends on 7)

The electronic device of claim 7, wherein the location information corresponding to the representation includes a plurality of boundary locations, each boundary location of the plurality of boundary locations corresponding to a respective physical boundary in the physical environment.

### Claim 10 (depends on 1)

The electronic device of claim 1, the one or more programs including instructions for:
in response to receiving the input including the reference to the first object, identifying the first object as a secondary electronic device;
after identifying the first object as the secondary device, receiving, from the secondary electronic device, a positional value corresponding to a location of the secondary electronic device within the physical environment; and
determining the location of the first object based on the received positional value.

### Claim 11 (depends on 1)

The electronic device of claim 1, the one or more programs including instructions for:
obtaining spatial information corresponding to the physical environment of the electronic device, wherein the spatial information is based on a multi-dimensional representation of the environment;
obtaining a positional value corresponding to the location of the first object; and
in accordance with a determination that the positional value corresponding to the location of the first object is included within the spatial information corresponding to the physical environment, determining that the location of the first object is available.

### Claim 12 (depends on 11)

The electronic device of claim 11, the one or more programs including instructions for:
in accordance with a determination that the positional value corresponding to the location of the first object is not included within the spatial information corresponding to the physical environment, determining that the location of the first object is not available.

### Claim 13 (depends on 1)

The electronic device of claim 1, the one or more programs including instructions for:
obtaining a multi-dimensional representation of the physical environment, wherein the multi-dimensional representation of the physical environment includes positional values corresponding to a plurality of object representations and positional values corresponding to a plurality of boundary representations, wherein the one or more first criteria and the one or more second criteria are based on the obtained multi-dimensional representation.

### Claim 14 (depends on 1)

The electronic device of claim 1, wherein the location of the first object is determined in response to receiving the input including the reference to the first object.

### Claim 15 (depends on 1)

The electronic device of claim 1, the one or more programs including instructions for:
in accordance with a determination that the first object satisfies one or more third criteria, periodically obtaining the positional value corresponding to the location of the first object.

### Claim 16 (depends on 1)

The electronic device of claim 1, wherein the reference within the physical environment includes an identified area within the physical environment, and wherein the identified area is associated with at least one area type.

### Claim 17 (depends on 1)

The electronic device of claim 1, wherein the reference within the physical environment includes a second object identified within the physical environment, wherein the second object is associated with at least one object type.

### Claim 18 (depends on 1)

The electronic device of claim 1, wherein the reference within the physical environment includes an identified spatial relation between a second object identified within the physical environment and a third object identified within the physical environment.

### Claim 19

A computer-implemented method, comprising:
at an electronic device with one or more processors and memory:
receiving an input including a reference to a first object, wherein
the input includes a request for a location of the first object, and
the first object is located in a physical environment;
determining a location of the first object; and
in response to receiving the input:
in accordance with a determination that one or more first criteria are met, providing a description of the location of the first object, wherein
a criterion of the one or more first criteria is met when the location of the first object is available, and
the description includes a relationship between the first object and a reference within the physical environment; and
in accordance with a determination that one or more second criteria are met, forgoing providing the description of the location of the first object, wherein a criterion of the one or more second criteria is met when the location of the first object is not available.

### Claim 20

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of a first electronic device, cause the first electronic device to:
receive an input including a reference to a first object, wherein
the input includes a request for a location of the first object, and
the first object is located in a physical environment;
determine a location of the first object; and
in response to receiving the input:
in accordance with a determination that one or more first criteria are met, provide a description of the location of the first object, wherein
a criterion of the one or more first criteria is met when the location of the first object is available, and
the description includes a relationship between the first object and a reference within the physical environment; and
in accordance with a determination that one or more second criteria are met, forgo providing the description of the location of the first object, wherein a criterion of the one or more second criteria is met when the location of the first object is not available.

### Claim 21

A electronic device, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving, from the image sensor, image data corresponding to an environment of the electronic device;
obtaining a representation of the environment, wherein the representation is generated based on the image data;
identifying a location of a first object within the environment, wherein the first object corresponds to a secondary electronic device;
providing a description of the location of the first object; and
in response to a user selection of the provided description:
adding a representation of the first object to the representation; and
assigning the selected description to the first object.

### Claim 22 (depends on 21)

The electronic device of claim 21, wherein obtaining a representation of the environment comprises:
obtaining spatial information including a plurality of object representations and a plurality of boundary representations.

### Claim 23 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
in accordance with a determination that the representation of the environment includes an object representation corresponding to a predetermined object type:
identifying an area within the representation of the environment, wherein the object representation is positioned within the identified area; and
assigning, based on the predetermined object type, an area type to the identified area, wherein the selected description includes the area type.

### Claim 24 (depends on 21)

The electronic device of claim 21, wherein identifying a location of the first object within the environment comprises:
displaying the representation of the environment, wherein the displayed representation includes a multi-dimensional representation of the environment; and
detecting a user selection of a location within the displayed representation of the environment.

### Claim 25 (depends on 21)

The electronic device of claim 21, wherein identifying a location of the first object within the environment comprises:
detecting a wireless signal transmitted from a third electronic device located within the environment of the electronic device; and
identifying the location of the first object based on a strength of the wireless signal.

### Claim 26 (depends on 25)

The electronic device of claim 25, wherein the third electronic device corresponds to a wireless router.

### Claim 27 (depends on 21)

The electronic device of claim 21, wherein identifying a location of the first object within the environment comprises:
detecting a wireless signal transmitted from a third electronic device located within the environment of the electronic device; and
identifying the location of the first object based on timing information corresponding to the transmitted wireless signal.

### Claim 28 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
in response to a user rejection of the provided description:
receiving, responsive to a provided prompt, a device description from the user;
adding a representation of the first object to the representation of the environment; and
assigning the received device description to the first object.

### Claim 29 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
providing the description of the location of the first object by outputting an audible prompt including the description; and
receiving speech input from a user of the electronic device, wherein the received speech input corresponds to the user selection of the provided description.

### Claim 30 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
providing the description of the location of the first object by displaying a textual representation of the description on a display of the electronic device; and
receiving a touch input from a user of the electronic device, wherein the received touch input corresponds to the user selection of the provided description.

### Claim 31 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
while receiving the image data corresponding to the environment of the electronic device:
displaying captured image data from the image sensor; and
displaying an instruction to move the electronic device to capture additional image data.

### Claim 32 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
while receiving the image data corresponding to the environment of the electronic device:
obtaining a plurality of partial representations of the environment; and
displaying captured image data from the image sensor, wherein the displayed image data includes a visual indication whether a partial representation of the environment corresponding to the image data has been obtained.

### Claim 33 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
receiving an input including a request for a location of a respective object; and
providing a description of the location of the referenced object based on the obtained representation of the environment.

### Claim 34 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
while receiving the image data corresponding to the environment of the electronic device:
in response to a determination that an amount of area corresponding to received image data exceeds a threshold amount of area, displaying, as a multi-dimensional representation, the representation of the environment.

### Claim 35 (depends on 21)

The electronic device of claim 21, the one or more programs including instructions for:
periodically identifying a location of a respective object within the environment, wherein the periodic identification includes:
identifying a first location at a first time; and
identifying a second location at a second time, wherein the first location is different from the second location; and
in response to each identification of the location of the respective object within the environment, updating the representation of the environment to include the most recent identified location of the respective object.

### Claim 36

A computer-implemented method, the one or more programs including instructions for:
at an electronic device with one or more processors, memory, and an image sensor:
receiving, from the image sensor, image data corresponding to an environment of the electronic device;
obtaining a representation of the environment, wherein the representation is generated based on the image data;
identifying a location of a first object within the environment, wherein the first object corresponds to a secondary electronic device;
providing a description of the location of the first object; and
in response to a user selection of the provided description:
adding a representation of the first object to the representation; and
assigning the selected description to the first object.

### Claim 37

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of a first electronic device, cause the electronic device to:
receive, from the image sensor, image data corresponding to an environment of the electronic device;
obtain a representation of the environment, wherein the representation is generated based on the image data;
identify a location of a first object within the environment, wherein the first object corresponds to a secondary electronic device;
provide a description of the location of the first object; and
in response to a user selection of the provided description:
add a representation of the first object to the representation; and
assign the selected description to the first object.
