# US 2023/0410440 A1 · Integrating augmented reality experiences with other components

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410440 A1
- Document ID: US20230410440A1
- Status: Published application
- Kind: A1
- Application number: 17/845,496
- Filed: 2022-06-21
- Published: 2023-12-21
- Earliest claimed: 2022-06-21
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Rastan Boroujerdi; Michael John Evans; Panayoti Haritatos
- Web page: https://patentgrep.com/patent/US20230410440A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04L 51/10: Multimedia information
- G01C 21/3647: {Guidance involving output of stored or live camera images or video streams}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- H04L 51/046: Interoperability with other network applications or services
- H04L 63/083: {using passwords (cryptographic mechanisms or cryptographic arrangements for entity authentication using a predetermined code H04L9/3226)}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2219/024: Multi-user, collaborative environment
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems are disclosed for generating AR experiences. The methods and systems access a first component of a plurality of components implemented by the messaging application, the plurality of components comprising an AR experience, each of the plurality of components being configured to be separately launched by the messaging application. The methods and systems store a first state of the first component in a data structure that is shared across the plurality of components; launching. The methods and system launch, by the messaging application, a second component of the plurality of components in response to determining that an interaction has been performed using the first component; and configure a second state of the second component based on the interaction that has been performed using the first component.

## Claims

### Claim 1

A method comprising:
accessing, by a messaging application implemented on a client device, a first component of a plurality of components implemented by the messaging application, the plurality of components comprising an augmented reality (AR) experience, each of the plurality of components being configured to be separately launched by the messaging application;
storing a first state of the first component in a data structure that is shared across the plurality of components;
launching, by the messaging application, a second component of the plurality of components in response to determining that an interaction has been performed using the first component; and
configuring a second state of the second component based on the interaction that has been performed using the first component.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first component comprises the AR experience, and wherein the second component comprises at least one of a web-based application, a map application, or an avatar application.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second component comprises the AR experience, and wherein the first component comprises at least one of a web-based application, a map application, or an avatar application.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining that the interaction has been performed using the first component comprises:
determining that input has been received from a user corresponding to a specified gesture; and
executing, by the first component, a call to action operation associated with the second component to launch the second component.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
in response to determining that the input has been received corresponding to the specified gesture, displaying by the first component a prompt with an option to execute the call to action operation; and
receiving a second input that selects the option to execute the call to action operation.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining that the interaction has been performed using the first component comprises:
determining that a goal has been reached during use of the first component; and
in response to determining that the goal has been reached, automatically executing, by the first component, a call to action operation associated with the second component to launch the second component.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
replacing a display of the first component with a display of the second component in response to launching the second component.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
detecting a specified interaction performed using the second component;
updating the data structure to represent a current state of the second component; and
re-accessing the first component based on the updated data structure in response to detecting the specified interaction performed using the second component, wherein the display of the second component is replaced with the display of the first component after the first component is re-accessed.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
detecting a specified interaction performed using the second component;
updating the data structure to represent a current state of the second component; and
launching, by the second component, a third component of the plurality of components based on the updated data structure in response to detecting the specified interaction performed using the second component, wherein the display of the second component is replaced with a display of the third component after the third component is launched.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first component comprises the AR experience and the second component comprises a web-based application, the state of the first component comprises a payload of parameters of the AR experience which is used to adjust one or more parameters of the second component to provide a unified multi-component experience to a user.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
generating for display a first user interface associated with the AR experience in response to receiving a user selection of an option to separately launch the second component comprising the AR experience; and
generating for display a second user interface associated with the AR experience in response to launching the second component in response to the interaction performed using the first component, wherein one or more options available in the first user interface are disabled in the second user interface.

### Claim 12 (depends on 1)

The method of claim 1, wherein an integration between the plurality of components comprising at least the AR experience is developed using an AR developer platform, further comprising accessing, by the second component, the data structure to retrieve the first state of the first component and to configure the state of the second component.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first component comprises the AR experience, and the second component comprises a map application, further comprising:
applying, by the first component, one or more AR elements to a real-world object depicted in an image captured by a camera of the client device, the real-world object being associated with a real-world location;
storing data representing the one or more AR elements and the real-world location as the first state in the data structure;
navigating a map displayed by the map application to a region comprising the real-world location; and
modifying a portion of the region of the map corresponding to the real-world object to include the one or more AR elements.

### Claim 14 (depends on 13)

The method of claim 13, wherein the client device comprises a first client device associated with a first user, wherein the first component comprising the AR experience is accessed by the first client device, and wherein the map application is accessed by a second client device to display the modified portion of the region of the map on the second client device associated with a second user.

### Claim 15 (depends on 1)

The method of claim 1, wherein the first component comprises a web-based application, and wherein the second component comprises the AR experience, further comprising:
navigating an avatar in the web-based application to a virtual location;
determining that the virtual location is associated with the AR experience comprising activation of a front-facing camera of the client device;
in response to navigating the avatar in the web-based application to the virtual location, adding one or more AR elements of the AR experience to a real-time video feed captured by the front-facing camera of the client device; and
presenting the real-time video feed that includes the one or more AR elements within a display of the web-based application.

### Claim 16 (depends on 1)

The method of claim 1, wherein the first component comprises a web-based application, and wherein the second component comprises the AR experience, further comprising:
receiving input by the web-based application comprising selection of a virtual item;
determining that the selection of the virtual item is associated with the AR experience comprising activation of a rear-facing camera of the client device;
in response to receiving the input by the web-based application comprising selection of the virtual item, adding one or more AR elements of the AR experience representing the virtual item to a real-time video feed captured by the rear-facing camera of the client device; and
presenting the real-time video feed that includes the one or more AR elements representing the virtual item of the web-based application.

### Claim 17 (depends on 1)

The method of claim 1, wherein the first component comprises the AR experience, and wherein the second component comprises a web-based application, further comprising:
presenting, by the AR experience, a first AR element comprising a virtual device;
in response to receiving input by the AR experience to unlock the virtual device, causing the web-based application to present a keypad;
receiving, by the web-based application, a request for a passcode hint;
in response to receiving the request for the passcode hint, causing the AR experience to present one or more AR elements representing the passcode hint over a real-time video feed captured by a camera of the client device;
receiving, by the web-based application, interaction with the keypad comprising a correct passcode associated with the passcode hint represented by the one or more AR elements; and
enabling, by the web-based application, a user to interact with the virtual device in response to receiving the interaction with the keypad.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
performing one or more interactions with the AR experience after access to the virtual device has been enabled;
determining, by the AR experience, that the one or more interactions correspond to a goal; and
in response to determining that the one or more interactions correspond to the goal, unlocking access, by the web-based application, to other features of the virtual device.

### Claim 19

A system comprising:
a processor; and
a memory component having instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
accessing, by a messaging application implemented on a client device, a first component of a plurality of components implemented by the messaging application, the plurality of components comprising an augmented reality (AR) experience, each of the plurality of components being configured to be separately launched by the messaging application;
storing a first state of the first component in a data structure that is shared across the plurality of components;
launching, by the messaging application, a second component of the plurality of components in response to determining that an interaction has been performed using the first component; and
configuring a second state of the second component based on the interaction that has been performed using the first component.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations comprising:
accessing, by a messaging application implemented on a client device, a first component of a plurality of components implemented by the messaging application, the plurality of components comprising an augmented reality (AR) experience, each of the plurality of components being configured to be separately launched by the messaging application;
storing a first state of the first component in a data structure that is shared across the plurality of components;
launching, by the messaging application, a second component of the plurality of components in response to determining that an interaction has been performed using the first component; and
configuring a second state of the second component based on the interaction that has been performed using the first component.
