# US 2024/0202944 A1 · Aligning scanned environments for multi-user communication sessions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202944 A1
- Document ID: US20240202944A1
- Status: Published application
- Kind: A1
- Application number: 18/592,586
- Filed: 2024-03-01
- Published: 2024-06-20
- Earliest claimed: 2021-09-16
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Bruno M. SOMMER; Alexandre DA VEIGA; Long H. NGO; Sebastian P. HERSCHER
- Web page: https://patentgrep.com/patent/US20240202944A1

## CPC

- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- G06T 7/337: {involving reference images or patches}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/56: Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
- A63F 13/655: by importing photos, e.g. of the player
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/74: {involving reference images or patches}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2219/024: Multi-user, collaborative environment
- G06T 2219/2004: Aligning objects, relative positioning of parts
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Various implementations provide a method for determining position data of a first device relative to a three-dimensional (3D) representation during a communication session. For example, a 3D representation is determined by a first device to correspond to a current physical environment of the first device. Then a spatial relationship is determined between the 3D representation and the current physical environment. Then position data is determined to correspond to a position of the first device relative to the 3D representation and based on a location of the first device in the current physical environment and the spatial relationship between the 3D representation and the current physical environment. The position data is then provided during a communication session between the first device and a second device with a view of the 3D representation including a representation of a user of the first device presented to a user of the second device.

## Claims

### Claim 1

A method comprising:
at a first device comprising one or more processors:
determining that a three-dimensional (3D) representation of an environment corresponds to a current physical environment of the first device;
determining a spatial relationship between the 3D representation and the current physical environment;
determining position data corresponding to a position of the first device relative to the 3D representation based on a location of the first device in the current physical environment and the spatial relationship between the 3D representation and the current physical environment; and
providing the position data during a communication session between the first device and a second device, wherein a view of the 3D representation including a representation of a user of the first device positioned based on the position data is presented to a user of the second device during the communication session.

### Claim 2 (depends on 1)

The method of claim 1, wherein the 3D representation comprises a point cloud representing at least a portion of a physical environment.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the spatial relationship between the 3D representation and the current physical environment comprises aligning features based on visual matching.

### Claim 4 (depends on 3)

The method of claim 3, wherein the 3D representation comprises a feature map that includes features associated with a physical environment, and the visual matching is based on matching the feature map associated with the 3D representation with features identified in the current physical environment.

### Claim 5 (depends on 3)

The method of claim 3, wherein the 3D representation comprises anchor points, each anchor point associated with a particular location within a physical environment, and the visual matching is based on the anchor points.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the spatial relationship between the 3D representation and the current physical environment is based on visual simultaneous localization and mapping (SLAM).

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the spatial relationship between the 3D representation and the current physical environment is based on matching a WiFi signature of the 3D representation and a WiFi signature of the first device in the current physical environment.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the spatial relationship between the 3D representation and the current physical environment is based on a best fit analysis between the 3D representation and sensor data of the first device in the current physical environment.

### Claim 9 (depends on 1)

The method of claim 1, wherein determining the spatial relationship between the 3D representation and the current physical environment is based on matching semantic data associated with objects in a physical environment represented by the 3D representation and objects in the current physical environment.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first device and the second device are in different physical environments.

### Claim 11 (depends on 1)

The method of claim 1, wherein the representation of at least a portion of the user of the first device is generated based on sensor data obtained during the communication session.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
updating at least a portion of the view at the second device based on user input at the first device.

### Claim 13 (depends on 1)

The method of claim 1, wherein the view of the 3D representation at the first device comprises an indication of the view the 3D representation at the second device.

### Claim 14 (depends on 1)

The method of claim 1, wherein a view of the communication session is presented in an extended reality (XR) experience.

### Claim 15 (depends on 1)

The method of claim 1, wherein the first device or the second device is a head-mounted device (HMD).

### Claim 16

A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
determining that a three-dimensional (3D) representation corresponds to a current physical environment of a first device;
determining a spatial relationship between the 3D representation and the current physical environment;
determining position data corresponding to a position of the first device relative to the 3D representation, based on a location of the first device in the current physical environment and the spatial relationship between the 3D representation and the current physical environment; and
providing the position data during a communication session between the first device and a second device, wherein a view of the 3D representation including a representation of a user of the first device positioned based on the position data is presented to a user of the second device during the communication session.

### Claim 17 (depends on 16)

The system of claim 16, wherein determining the spatial relationship between the 3D representation with the current physical environment comprises aligning features based on visual matching.

### Claim 18 (depends on 17)

The system of claim 17, wherein the 3D representation comprises a feature map that includes features associated with the current physical environment, and the visual matching is based on matching the feature map associated with the 3D representation with features identified in the current physical environment.

### Claim 19 (depends on 17)

The system of claim 17, wherein the 3D representation comprises anchor points, each anchor point associated with a particular location within the current physical environment, and the visual matching is based on the anchor points.

### Claim 20

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
determining that a three-dimensional (3D) representation corresponds to a current physical environment of a first device;
determining a spatial relationship between the 3D representation and the current physical environment;
determining position data corresponding to a position of the first device relative to the 3D representation, based on a location of the first device in the current physical environment and the spatial relationship between the 3D representation and the current physical environment; and
providing the position data during a communication session between the first device and a second device, wherein a view of the 3D representation including a representation of a user of the first device positioned based on the position data is presented to a user of the second device during the communication session.

### Claim at position 21

21-43. (canceled)
