# US 11,908,159 B2 · Systems and methods for representing user interactions in multi-user augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,159 B2
- Document ID: US11908159B2
- Status: Granted
- Kind: B2
- Application number: 16/939,280
- Filed: 2020-07-27
- Published: 2022-01-27
- Granted: 2024-02-20
- Earliest claimed: 2020-07-27
- Assignees: Shopify Inc.
- Applicants: Shopify Inc.
- Inventors: Juho Mikko Haapoja; Byron Leonel Delgado; Stephan Leroux; Daniel Beauchamp
- Web page: https://patentgrep.com/patent/US11908159B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/73: using feature-based methods
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 2219/024: Multi-user, collaborative environment
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods are provided for the generation of augmented reality (AR) content that provides a shared AR experience involving multiple users. Shared AR experiences can improve the communication and collaboration between multiple simultaneous users. According to an embodiment, AR content is generated for a first user in a shared AR experience. The AR content includes at least one of a render of a model, a virtual representation of a second user in the shared AR experience, a virtual representation of a user interaction in the shared AR experience, and spatial audio content. Modifications to a shared AR experience are also provided. These modifications may be initiated based on instructions from one user and be reflected in the AR content generated for multiple users.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining a position and orientation of a hand of a first user, the hand interacting with virtual content presented at a first device, the virtual content comprising a first render of a virtual 3D (three-dimensional) model defined within a virtual coordinate system, the position and the orientation of the hand being defined within the virtual coordinate system;
generating augmented reality (AR) content for presentation at a second device overlaid onto a view of a real-world space, the AR content comprising both:
a second render of the virtual 3D model based on a position of a second user within the virtual coordinate system, the position of the second user being based on information obtained by the second device representing the real-world space, and
a virtual representation of the hand of the first user interacting with the second render of the virtual 3D model by displaying the position and orientation of the hand of the first user in relation to the second render of the virtual 3D model, the virtual representation and the second render of the virtual 3D model both overlaid onto the view of the real-world space; and
outputting the AR content for presentation at the second device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining a position of the first user within the virtual coordinate system,
wherein the position of the hand is based on the position of the first user within the virtual coordinate system.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining, using the first device, a representation of a space,
wherein obtaining the position of the hand of the first user within the virtual coordinate system comprises:
determining a position of the virtual 3D model within the space,
mapping the virtual coordinate system to the space based on the position of the virtual 3D model within the space, and
determining a position of the hand of the first user within the space, wherein the position of the hand of the first user within the virtual coordinate system is based on the position of the hand of the first user within the space.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein:
the representation of the space comprises an image of the space captured by the first device; and
determining the position of the hand of the first user within the space is based on image analysis of the image.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein:
the AR content is first AR content;
the real-world space is a first real-word space; and
the virtual content comprises second AR content for presentation at the first device overlaid onto a view of a second real-world space.

### Claim 6

A system comprising:
memory to store a position and orientation of a hand of a first user, the hand interacting with virtual content presented at a first device, the virtual content comprising a first render of a virtual three-dimensional (3D) model defined within a virtual coordinate system, the position and orientation of the hand being defined within the virtual coordinate system; and
at least one processor configured to:
generate augmented reality (AR) content for presentation at a second device overlaid onto a view of a real-world space, the AR content comprising both:
a second render of the virtual 3D model based on a position of a second user within the virtual coordinate system, the position of the second user being based on information obtained by the second device representing the real-world space, and
a virtual representation of the hand of the first user interacting with the second render of the virtual 3D model by displaying the position and orientation of the hand of the first user in relation to the second render of the virtual 3D, the virtual representation and the second render of the virtual 3D model both overlaid onto the view of the real-world space; and
output the AR content for presentation at the second device.

### Claim 7 (depends on 6)

The system of claim 6, wherein the at least one processor is further configured to obtain a position of the first user within the virtual coordinate system, wherein the position of the hand is based on the position of the first user within the virtual coordinate system.

### Claim 8 (depends on 6)

The system of claim 6, wherein the at least one processor is further configured to:
obtain, using the first device, a representation of a space,
determine a position of the virtual 3D model within the space,
map the virtual coordinate system to the space based on the position of the virtual 3D model within the space, and
determine a position of the hand of the first user within the space, wherein the position of the hand of the first user within the virtual coordinate system is based on the position of the hand of the first user within the space.

### Claim 9 (depends on 8)

The system of claim 8, wherein:
the representation of the space comprises an image of the space captured by the first device; and
the position of the hand of the first user within the space is based on image analysis of the image.

### Claim 10 (depends on 6)

The system of claim 6, wherein:
the AR content is first AR content;
the real-world space is a first real-word space; and
the virtual content comprises second AR content for presentation at the first device overlaid onto a view of a second real-world space.

### Claim 11

A non-transitory computer readable medium storing computer executable instructions which, when executed by a computer, cause the computer to:
obtain a position and orientation of a hand of a first user, the hand interacting with virtual content presented at a first device, the virtual content comprising a first render of a virtual three-dimensional (3D) model defined within a virtual coordinate system, the position and orientation of the hand being defined within the virtual coordinate system;
generate augmented reality (AR) content for presentation at a second device overlaid onto a view of a real-world space, the AR content comprising both:
a second render of the virtual 3D model based on a position of a second user within the virtual coordinate system, the position of the second user being based on information obtained by the second device representing the real-world space, and
a virtual representation of the hand of the first user interacting with the second render of the virtual 3D model by displaying the position and orientation of the hand of the first user in relation to the second render of the virtual 3D model, the virtual representation and the second render of the virtual 3D model both overlaid onto the view of the real-world space; and
output the AR content for presentation at the second device.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the virtual content is overlaid onto a virtual reality (VR) environment.

### Claim 13 (depends on 6)

The system of claim 6, wherein the virtual content is overlaid onto a virtual reality (VR) environment.

### Claim 14 (depends on 3)

The computer-implemented method of claim 3, wherein the representation of the space is based on a previously generated representation of the space including measurements previously obtained.

### Claim 15 (depends on 8)

The system of claim 8, wherein the representation of the space is based on a previously generated representation of the space including measurements previously obtained.

### Claim 16 (depends on 3)

The computer-implemented method of claim 3, wherein:
the first user is wearing an AR-specific glove; and
determining the position of the hand of the first user within the space is based a position of the AR-specific glove in the space.

### Claim 17 (depends on 8)

The system of claim 8, wherein:
the first user is wearing an AR-specific glove; and
the position of the hand of the first user within the space is based a position of the AR-specific glove in the space.

### Claim 18 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the instructions, when executed by the computer, further cause the computer to:
obtain, using the first device, a representation of a space,
determine a position of the virtual 3D model within the space,
map the virtual coordinate system to the space based on the position of the virtual 3D model within the space, and
determine a position of the hand of the first user within the space, wherein the position of the hand of the first user within the virtual coordinate system is based on the position of the hand of the first user within the space.

### Claim 19 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein:
the representation of the space comprises an image of the space captured by the first device; and
the position of the hand of the first user within the space is based on image analysis of the image.

### Claim 20 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the representation of the space is based on a previously generated representation of the space including measurements previously obtained.

### Claim 21 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein:
the first user is wearing an AR-specific glove; and
the position of the hand of the first user within the space is based a position of the AR-specific glove in the space.

### Claim 22 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the instructions, when executed by the computer, further cause the computer to:
obtain a position of the first user within the virtual coordinate system, wherein the position of the hand is based on the position of the first user within the virtual coordinate system.

### Claim 23 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein:
the AR content is first AR content;
the real-world space is a first real-word space; and
the virtual content comprises second AR content for presentation at the first device overlaid onto a view of a second real-world space.