# US 2025/0166322 A1 · Dynamic Mixed Reality Content In Virtual Reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166322 A1
- Document ID: US20250166322A1
- Status: Published application
- Kind: A1
- Application number: 19/033,911
- Filed: 2025-01-22
- Published: 2025-05-22
- Earliest claimed: 2021-06-02
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Sarah Tanner Simpson; Gregory Smith; Jeffrey Witthuhn; Ying-Chieh Huang; Shuang Li; Wenliang Zhao; Peter Koch; Meghana Reddy Guduru; Ioannis Pavlidis; Xiang Wei; Kevin Xiao; Kevin Joseph Sheridan; Bodhi Donselaar; Federico Adrian Camposeco Paulsen
- Web page: https://patentgrep.com/patent/US20250166322A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity
- A63F 13/235: using a wireless connection, e.g. infrared or piconet
- A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- A63F 13/92: Video game devices specially adapted to be hand-held while playing
- G06F 3/012: {Head tracking input arrangements}
- G06T 7/11: Region-based segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/20: in augmented reality scenes
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06T 2207/30196: Human being; Person
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one embodiment, a method includes capturing images of a first user wearing a VR display device in a real-world environment. The method includes receiving a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes generating a first MR rendering of the first user in the VR environment., The first MR rendering of the first user is based on a compositing of the images of the first user and the VR rendering. The method includes receiving an indication of a user interaction with one or more elements of the VR environment in the first MR rendering. The method includes generating, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment. The one or more elements are modified according to the interaction.

## Claims

### Claim 1

A method comprising:
capturing one or more images of a user wearing an artificial reality (XR) display device;
receiving, from the XR display device, a first rendering of an environment, wherein the first rendering is from the perspective of a capture device with respect to the XR display device;
generating, responsive to capturing the one or more images, a second rendering depicting the user in the environment, wherein the second rendering is based on a compositing of the one or more images and the first rendering;
receiving an indication of a change in the environment; and
generating, responsive to the indication of the change, a third rendering depicting the user in the environment and the change in the environment.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving one or more anchor points, in the real-world environment, identified from the one or more images.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining a pose of the capture device with respect to the XR display device based on one or more of the anchor points.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
detecting the capture device has moved;
determining an updated pose of the capture device with respect to the XR display device based on one or more of the anchor points; and
generating, responsive to detecting the capture device has moved, a fourth rendering of the user in the environment, wherein the fourth rendering of the user is based on the updated pose of the capture device with respect to the XR display device.

### Claim 5 (depends on 3)

The method of claim 3, wherein the pose of the capture device with respect to the XR display device is determined based on a triangulation between one or more of the anchor points, the one or more cameras of the capture device, and the XR display device.

### Claim 6 (depends on 2)

The method of claim 2, wherein the one or more anchor points are fixed objects in the real-world environment.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
toggling between:
an XR rendering of the environment from the perspective of the user, and
a rendering of the user in the environment from the perspective of the capture device with respect to the XR display device.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
transmitting the second or third rendering of the user in the environment to an online social network for live display to one or more additional users.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
segmenting a representation of the user from the one or more images; and
identifying, using a segmentation algorithm, a body of the user within each of the one or more images, wherein the second or third rendering of the user is based on a compositing of the segmented one or more images of the user and the first rendering.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving an indication the user has activated a particular XR application; and
presenting, responsive to the received indication, a notification that the user wearing the XR display device has started the particular XR application.

### Claim 11 (depends on 1)

The method of claim 1, wherein the indication of the change in the environment is received on an interface of the capture device.

### Claim 12 (depends on 1)

The method of claim 1, wherein the change in the environment comprises one or more of:
a modification of an object in the environment,
creation of an object in the environment,
a change of the environment from a first environment to a second environment; or
creation of a first avatar corresponding to a second user, or an interaction with a second avatar of a third user.

### Claim 13

A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to:
capture one or more images of a user wearing an artificial reality (XR) display device;
receive, from the XR display device, a first rendering of an environment, wherein the first rendering is from the perspective of a capture device with respect to the XR display device;
generate, responsive to capturing the one or more images, a second rendering depicting the user in the environment, wherein the second rendering is based on a compositing of the one or more images and the first rendering;
receive an indication of a change in the environment; and
generate, responsive to the indication of the change, a third rendering depicting the user in the environment and the change in the environment.

### Claim 14 (depends on 13)

The computer-readable storage medium of claim 13, wherein the instructions, when executed, further cause the computing system to:
receive one or more anchor points, in the real-world environment, identified from the one or more images.

### Claim 15 (depends on 14)

The computer-readable storage medium of claim 14, wherein the instructions, when executed, further cause the computing system to:
determine a pose of the capture device with respect to the XR display device based on one or more of the anchor points.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the instructions, when executed, further case the computing system to:
detect the capture device has moved;
determine an updated pose of the capture device with respect to the XR display device based on one or more of the anchor points; and
generate, responsive to detecting the capture device has moved, a fourth rendering of the user in the environment, wherein the fourth rendering of the user is based on the updated pose of the capture device with respect to the XR display device.

### Claim 17 (depends on 15)

The computer-readable storage medium of claim 15, wherein the pose of the capture device with respect to the XR display device is determined based on a triangulation between one or more of the anchor points, the one or more cameras of the capture device, and the XR display device.

### Claim 18 (depends on 13)

The computer-readable storage medium of claim 13, wherein the instructions, when executed, further case the computing system to:
transmit the second or third rendering of the user in the environment to an online social network for live display to one or more additional users.

### Claim 19 (depends on 13)

The computer-readable storage medium of claim 13, wherein the instructions, when executed, further case the computing system to:
segment a representation of the user from the one or more images; and
identify, using a segmentation algorithm, a body of the user within each of the one or more images, wherein the second or third rendering of the user is based on a compositing of the segmented one or more images of the user and the first rendering.

### Claim 20

A computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to:
capture one or more images of a user wearing an artificial reality (XR) display device;
receive, from the XR display device, a first rendering of an environment, wherein the first rendering is from the perspective of a capture device with respect to the XR display device;
generate, responsive to capturing the one or more images, a second rendering depicting the user in the environment, wherein the second rendering is based on a compositing of the one or more images and the first rendering;
receive an indication of a change in the environment; and
generate, responsive to the indication of the change, a third rendering depicting the user in the environment and the change in the environment.
