# US 2021/0241529 A1 · Augmented reality session creation using skeleton tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241529 A1
- Document ID: US20210241529A1
- Status: Published application
- Kind: A1
- Application number: 16/782,874
- Filed: 2020-02-05
- Published: 2021-08-05
- Earliest claimed: 2020-02-05
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Piers Cowburn; David Li; Isac Andreas Müller Sandvik; Qi Pan; Matan Zohar
- Web page: https://patentgrep.com/patent/US20210241529A1

## CPC

- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/22: Setup operations, e.g. calibration, key configuration or button assignment
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/75: {involving models}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- H04L 65/1069: Session establishment or de-establishment
- H04L 65/1089: by adding media; by removing media
- A63F 13/48: Starting a game, e.g. activating a game device or waiting for other players to join a multiplayer session
- G06T 2219/024: Multi-user, collaborative environment
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Methods and systems are disclosed for creating a shared augmented reality (AR) session. The methods and systems perform operations comprising: receiving, by a second device, a request to join an AR session initialized by a first device; in response to receiving the request, detecting a body corresponding to a user of the first device in one or more images captured by a camera of the second device; identifying a body part of the detected body corresponding to the user of the first device; determining, by the second device, a transformation in the AR session between the first device and the second device using the identified body part; and causing the AR session to be displayed by the second device based on the determined transformation.

## Claims

### Claim 1

A method comprising:
receiving, by a second device, a request to join an AR session initialized by a first device;
in response to receiving the request, detecting a body corresponding to a user of the first device in one or more images captured by a camera of the second device;
identifying a body part of the detected body corresponding to the user of the first device;
determining, by the second device, a transformation in the AR session between the first device and the second device using the identified body part; and
causing the AR session to be displayed by the second device based on the determined transformation.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
determining a common coordinate frame using the transformation; and
generating the AR session using the common coordinate frame.

### Claim 3 (depends on 1)

The method of claim 1, wherein the AR session is initialized by the first device by determining, by the first device, a first device transformation based on a first device current pose and a first device origin, wherein the first device origin is a first origin of a first coordinate frame tracked by the first device.

### Claim 4 (depends on 3)

The method of claim 3, further comprising determining, by the second device, a second device transformation in response to receiving the request to join the AR session of the first device, the second device transformation being based on a second device current pose and a second device origin, wherein the second device origin is a second origin of a second coordinate frame tracked by the second device.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
sending to the first device, from the second device, the transformation representing coordinates of AR content on the second device relative to a position of the first device.

### Claim 6 (depends on 5)

The method of claim 5, wherein determining, by the second device, the transformation between the first device and the second device comprises:
determining, by the second device, an offset based on the first device transformation, using a current pose of the second device and an origin of the first device.

### Claim 7 (depends on 1)

The method of claim 1, where identifying the body part further comprises:
processing the one or more images to identify a plurality of skeletal positions of the body; and
identifying the skeletal position from the plurality of skeletal positions corresponding to the body part.

### Claim 8 (depends on 7)

The method of claim 7, wherein the body part is a wrist and the skeletal position corresponding to the body part comprises a wrist joint.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
searching the one or more images for an image of the first user device;
determining that the first user device is being held by the first user in a right hand of the first user; and
selecting the wrist corresponding to the right hand as the body part based on determining that the first user device is being held by the first user in the right hand of the first user.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first device uses a first coordinate frame to display AR objects, wherein the second device uses a second coordinate frame to display AR objects, and wherein causing the AR session to be displayed by the first device and by the second device comprises:
displaying a common AR object in the AR session on the second device based on the second coordinate frame;
causing the first device to adjust the first coordinate frame of the first device based on the transformation in the AR session between the first device and the second device; and
causing the first device to display the common AR object in the AR session on the first device based on adjusting the first coordinate frame.

### Claim 11 (depends on 1)

The method of claim 1, wherein the transformation is a first transformation, and the method further comprises:
calculating a second transformation during the AR session when the body is detected in additional images captured subsequently to the one or more images.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
calculating the second transformation each time the body is detected during the AR session.

### Claim 13 (depends on 11)

The method of claim 11, wherein the second transformation is calculated at predetermined time intervals.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
comparing the first transformation value to the second transformation;
computing an error based on comparing the first transformation to the second transformation;
in response to determining that the error is less than a threshold, maintaining display of the AR session based on the first transformation.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
in response to determining that the error is equal to or greater than the threshold, causing the AR session to be displayed by the first device and by the second device based on the second transformation.

### Claim 16 (depends on 14)

The method of claim 14, selecting the threshold from a plurality of thresholds based on positioning of content within the AR session.

### Claim 17 (depends on 16)

The method of claim 16, wherein a first threshold of the plurality of thresholds corresponds to AR content that is positioned in relation to one or more real-world objects, a second threshold of the plurality of thresholds corresponds to AR content that is positioned independently of any real-world objects, and the first threshold is smaller than the second threshold.

### Claim 18 (depends on 1)

The method of claim 1, further comprising:
initializing the AR session using a given transformation between the first device and the second device that is computed based on a marker displayed on the first device; and
updating the given transformation using the AR session based on the transformation determined using the identified body part.

### Claim 19

A system comprising:
a processor; and
a memory component having instructions stored thereon, when executed by the processor, causes the processor to perform operations comprising:
receiving, by a second device, a request to join an AR session initialized by a first device;
in response to receiving the request, detecting a body corresponding to a user of the first device in one or more images captured by a camera of the second device;
identifying a body part of the detected body corresponding to the user of the first device;
determining, by the second device, a transformation in the AR session between the first device and the second device using the identified body part; and
causing the AR session to be displayed by the second device based on the determined transformation.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon, instructions when executed by a processor, causes the processor to perform operations comprising:
receiving, by a second device, a request to join an AR session initialized by a first device;
in response to receiving the request, detecting a body corresponding to a user of the first device in one or more images captured by a camera of the second device;
identifying a body part of the detected body corresponding to the user of the first device;
determining, by the second device, a transformation in the AR session between the first device and the second device using the identified body part; and
causing the AR session to be displayed by the second device based on the determined transformation.