# US 2025/0209754 A1 · Techniques for orienting a visual representation of a remote user based on physical landmarks within local physical surroundings of a user during a shared artificial-reality interaction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0209754 A1
- Document ID: US20250209754A1
- Status: Published application
- Kind: A1
- Application number: 18/980,947
- Filed: 2024-12-13
- Published: 2025-06-26
- Earliest claimed: 2023-12-20
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Diogo Cunha; Gioacchino Noris; Henrik Hegner Tomra Skovsgaard; Britta Hummel; Vasyl Baran
- Web page: https://patentgrep.com/patent/US20250209754A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06T 7/73: using feature-based methods
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 20/20: in augmented reality scenes
- H04N 7/15: Conference systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of performing a shared artificial-reality interaction using an artificial-reality headset is provided. The method includes obtaining local data about a local physical area in which a user is wearing an artificial-reality headset. The method further includes identifying, via the local data, a local orientation of the user relative to the local physical area. The method further includes obtaining remote data indicating a remote orientation of a remote user relative to a remote physical landmark within a remote physical area different than the local physical area. And the method further includes, based on the local data, the local orientation of the user, and the remote data, and presenting, at the artificial-reality headset, a visual representation of the remote user at a co-present position within the local physical area with a representative orientation relative to (i) the local physical landmark and (ii) the local orientation of the user.

## Claims

### Claim 1

A non-transitory computer-readable storage medium, comprising instructions that, when executed by one or more processors, cause the processors to:
obtain local data about a local physical area in which a user is wearing an artificial-reality (AR) headset;
identify, via the local data, a local orientation of the user relative to the local physical area;
obtain remote data indicating a remote orientation of a remote user relative to a remote physical landmark within a remote physical area different than the local physical area; and
based on the local data, the local orientation of the user, and the remote data:
identify a local physical landmark that satisfies similarity criteria with respect to the remote physical landmark, and
present, at the AR headset, a visual representation of the remote user at a co-present position within the local physical area with a representative orientation relative to (i) the local physical landmark and (ii) the local orientation of the user.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein:
the local physical landmark is identified based on determining that the local physical landmark satisfies one or more similarity criteria, the one or more similarity criteria based on one or more of:
comparing semantic properties of the local physical landmark to other semantic properties of the remote physical landmark;
a physical relationship between the local physical landmark and one or more local real-world objects in the local physical area; and/or
a location of the remote user or the user in relation to a real-world boundary within the local physical area or the remote physical area.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, further comprising instructions for causing the one or more processors to:
responsive to a different remote user joining a shared AR experience being performed by the user and the remote user:
obtain other remote data identifying another remote position of the different remote user relative to another remote physical landmark, positioned within another remote physical area in which the remote user is located; and
present within the local physical area, based on the local data and the other remote data, another visual representation of the different remote user.

### Claim 4 (depends on 3)

The non-transitory computer-readable storage medium of claim 3, further comprising instructions for causing the one or more processors to:
in conjunction with presenting the other visual representation of the different remote user within the local physical area based on the other remote data:
cause an adjustment to the visual representation of the remote user that includes an interaction with the other visual representation of the different remote user.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, further comprising instructions for causing the one or more processors to:
based on at least the local data and the remote data obtained from the local physical area and the remote physical area:
generate a virtual scene, distinct from the local physical area and the remote physical area, wherein:
the virtual scene includes respective visual representations of the user and the remote user and at least one other visual representation corresponding to the local physical landmark and/or the remote physical landmark, and
the co-present position of the visual representation of the remote user is determined, at least in part, based on a virtual co-present position of the remote user within the virtual scene.

### Claim 6 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, further comprising instructions for causing the one or more processors to:
responsive to an input by the user to modify a presentation mode of the AR headset, such that the AR headset is caused to present a virtual-reality scene that encompasses substantially all of a particular field of view of the user:
present, at the AR headset, the virtual scene that includes the respective visual representations of the user and the remote user.

### Claim 7 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein:
the representative orientation of the visual representation of the remote user is based on a respective remote object, different than the remote physical landmark, that the remote user is interacting with in the remote physical area; and
the visual representation of the remote user is oriented to represent a remote-object orientation of the remote user with respect to the remote object.

### Claim 8 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the respective remote object is a physical object or a virtual object, and the non-transitory computer-readable storage medium further comprises instructions for causing the one or more processors to:
based on determining that the respective remote object is a physical object:
in accordance with a first determination that a respective local physical object satisfies similarity criteria with the remote object, orient the visual representation of the remote user based on a respective location of the respective local physical object;
in accordance with a second determination that no respective local physical object satisfies similarity criteria with respect to the remote object, and that a respective virtual object within the local physical area satisfies similarity criteria with respect to the remote object, orient the visual representation of the remote user based on another respective location of the respective virtual object;
in accordance with a third determination that no respective physical or virtual objects within the local physical area satisfy similarity criteria with respect to the remote object, perform one or mitigation operations, including:
prompting the user to cause a new virtual object to be presented within the local physical area, the new virtual object satisfying similarity criteria with respect to the remote object;
presenting a new virtual object satisfying similarity criteria with respect to the remote object, and orienting the visual representation of the remote user based on the new virtual object; and/or
forgoing orienting the visual representation of the remote user based on the remote object.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, further comprising instructions for causing the one or more processors to:
in accordance with the determining that no respective physical or virtual objects within the local physical area satisfy the similarity criteria with respect to the remote object, determine a mitigation operation to perform based on whether the remote object satisfies one or more co-present presentation criteria comprising:
the remote object includes an identifiable interactive portion for use as part of a physical interaction by the remote user,
the remote object is an interactive computing device having at least one user interface, and
the remote physical object is being used as part of a shared AR interaction between the user and the remote user.

### Claim 10

A system, comprising an augmented-reality headset, comprising one or more processors configured to:
obtain local data about a local physical area in which a user is wearing an artificial-reality (AR) headset;
identify, via the local data, a local orientation of the user relative to the local physical area;
obtain remote data indicating a remote orientation of a remote user relative to a remote physical landmark within a remote physical area different than the local physical area; and
based on the local data, the local orientation of the user, and the remote data:
identify a local physical landmark that satisfies similarity criteria with respect to the remote physical landmark, and
present, at the AR headset, a visual representation of the remote user at a co-present position within the local physical area with a representative orientation relative to (i) the local physical landmark and (ii) the local orientation of the user.

### Claim 11 (depends on 10)

The system of claim 10, wherein:
the local physical landmark is identified based on determining that the local physical landmark satisfies one or more similarity criteria, the one or more similarity criteria based on one or more of:
comparing semantic properties of the local physical landmark to other semantic properties of the remote physical landmark;
a physical relationship between the local physical landmark and one or more local real-world objects in the local physical area; and/or
a location of the remote user or the user in relation to a real-world boundary within the local physical area or the remote physical area.

### Claim 12 (depends on 10)

The system of claim 10, wherein the one or more processors are further configured to:
responsive to a different remote user joining a shared AR experience being performed by the user and the remote user:
obtain other remote data identifying another remote position of the different remote user relative to another remote physical landmark, positioned within another remote physical area in which the remote user is located; and
present within the local physical area, based on the local data and the other remote data, another visual representation of the different remote user.

### Claim 13 (depends on 12)

The system of claim 12, wherein the one or more processors are further configured to:
in conjunction with presenting the other visual representation of the different remote user within the local physical area based on the other remote data:
cause an adjustment to the visual representation of the remote user that includes an interaction with the other visual representation of the different remote user.

### Claim 14 (depends on 10)

The system of claim 10, wherein the one or more processors are further configured to:
based on at least the local data and the remote data obtained from the local physical area and the remote physical area:
generate a virtual scene, distinct from the local physical area and the remote physical area, wherein:
the virtual scene includes respective visual representations of the user and the remote user and at least one other visual representation corresponding to the local physical landmark and/or the remote physical landmark, and
the co-present position of the visual representation of the remote user is determined, at least in part, based on a virtual co-present position of the remote user within the virtual scene.

### Claim 15 (depends on 14)

The system of claim 14, wherein the one or more processors are further configured to:
responsive to an input by the user to modify a presentation mode of the AR headset, such that the AR headset is caused to present a virtual-reality scene that encompasses substantially all of a particular field of view of the user:
present, at the AR headset, the virtual scene that includes the respective visual representations of the user and the remote user.

### Claim 16 (depends on 10)

The system of claim 10, wherein:
the representative orientation of the visual representation of the remote user is based on a respective remote object, different than the remote physical landmark, that the remote user is interacting with in the remote physical area; and
the visual representation of the remote user is oriented to represent a remote-object orientation of the remote user with respect to the remote object.

### Claim 17 (depends on 16)

The system of claim 16, wherein the respective remote object is a physical object or a virtual object, and the one or more processors are further configured to:
based on determining that the respective remote object is a physical object:
in accordance with a first determination that a respective local physical object satisfies similarity criteria with the remote object, orient the visual representation of the remote user based on a respective location of the respective local physical object;
in accordance with a second determination that no respective local physical object satisfies similarity criteria with respect to the remote object, and that a respective virtual object within the local physical area satisfies similarity criteria with respect to the remote object, orient the visual representation of the remote user based on another respective location of the respective virtual object;
in accordance with a third determination that no respective physical or virtual objects within the local physical area satisfy similarity criteria with respect to the remote object, perform one or mitigation operations, including:
prompting the user to cause a new virtual object to be presented within the local physical area, the new virtual object satisfying similarity criteria with respect to the remote object;
presenting a new virtual object satisfying similarity criteria with respect to the remote object, and orienting the visual representation of the remote user based on the new virtual object; and/or
forgoing orienting the visual representation of the remote user based on the remote object.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more processors are further configured to:
in accordance with the determining that no respective physical or virtual objects within the local physical area satisfy the similarity criteria with respect to the remote object, determine a mitigation operation to perform based on whether the remote object satisfies one or more co-present presentation criteria comprising:
the remote object includes an identifiable interactive portion for use as part of a physical interaction by the remote user,
the remote object is an interactive computing device having at least one user interface, and
the remote physical object is being used as part of a shared AR interaction between the user and the remote user.

### Claim 19

A method, comprising:
obtain local data about a local physical area in which a user is wearing an artificial-reality (AR) headset;
identifying, via the local data, a local orientation of the user relative to the local physical area;
obtaining remote data indicating a remote orientation of a remote user relative to a remote physical landmark within a remote physical area different than the local physical area; and
based on the local data, the local orientation of the user, and the remote data:
identifying a local physical landmark that satisfies similarity criteria with respect to the remote physical landmark, and
presenting, at the AR headset, a visual representation of the remote user at a co-present position within the local physical area with a representative orientation relative to (i) the local physical landmark and (ii) the local orientation of the user.

### Claim 20 (depends on 19)

The method of claim 19, wherein:
the local physical landmark is identified based on determining that the local physical landmark satisfies one or more similarity criteria, the one or more similarity criteria based on one or more of:
comparing semantic properties of the local physical landmark to other semantic properties of the remote physical landmark;
a physical relationship between the local physical landmark and one or more local real-world objects in the local physical area; and/or
a location of the remote user or the user in relation to a real-world boundary within the local physical area or the remote physical area.
