# US 11,113,885 B1 · Real-time views of mixed-reality environments responsive to motion-capture data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,885 B1
- Document ID: US11113885B1
- Status: Granted
- Kind: B1
- Application number: 16/130,240
- Filed: 2018-09-13
- Granted: 2021-09-07
- Earliest claimed: 2017-09-13
- Assignees: Lucasfilm Entertainment Company Ltd.
- Applicants: Lucasfilm Entertainment Company Ltd.
- Inventors: Roger Cordes; David Brickhill
- Web page: https://patentgrep.com/patent/US11113885B1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/25: Output arrangements for video game devices
- A63F 13/35: Details of game servers
- A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
- A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- A63F 13/655: by importing photos, e.g. of the player
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/596: {from three or more stereo images}
- G06T 7/73: using feature-based methods
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 15/08: Volume rendering
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- H04L 65/611: for multicast or broadcast (systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06)
- G06T 2219/2021: Shape modification
- G06V 40/107: {Static hand or arm}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

An immersive content presentation system can capture the motion or position of a performer in a real-world environment. A game engine can be modified to receive the position or motion of the performer and identify predetermined gestures or positions that can be used to trigger actions in a 3-D virtual environment, such as generating a digital effect, transitioning virtual assets through an animation graph, adding new objects, and so forth. The use of the 3-D environment can be rendered and composited views can be generated. Information for constructing the composited views can be streamed to numerous display devices in many different physical locations using a customized communication protocol. Multiple real-world performers can interact with virtual objects through the game engine in a shared mixed-reality experience.

## Claims

### Claim 1

A method comprising:
receiving a motion or position of a first real-world performer in a first real-world environment;
identifying the motion or position as a predefined motion or position;
adding or altering a virtual asset in a 3-D virtual environment in response to identifying the motion or position as the predefined motion or position;
moving or positioning a virtual representation of the first real-world performer based on the motion or position of the first real-world performer;
rendering a 2-D video stream of the virtual asset and the virtual representation of the first real-world performer in the 3-D virtual environment; and
causing a real-time video stream to be visible on a display device of a handheld or wearable device, wherein:
the handheld or wearable device captures a real-time view of a second real-world environment comprising a second real-world performer;
the real-time video stream comprises the 2-D video stream of the virtual asset and the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; and
the motion or position of the first real-world performer in the first real-world environment adds or alters the virtual asset in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with the virtual asset when viewed on the display device of the handheld or wearable device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first real-world performer in the first real-world environment comprises a motion-capture suit that is recorded with a plurality of motion-capture cameras in a motion-capture system.

### Claim 3 (depends on 1)

The method of claim 1, wherein the motion or position of the first real-world performer is captured using a plurality of depth cameras.

### Claim 4 (depends on 1)

The method of claim 1, wherein the virtual asset comprises a CGI character.

### Claim 5 (depends on 4)

The method of claim 4, wherein adding or altering the virtual asset in the 3-D virtual environment comprises:
moving the CGI character according to the motion of the first real-world performer in the first real-world environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein the virtual asset comprises a digital effect that is added to the 3-D virtual environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein the virtual asset comprises an object that is added to the 3-D virtual environment.

### Claim 8

An immersive content presentation system comprising:
one or more processors; and
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a motion or position of a first real-world performer in a first real-world environment;
identifying the motion or position as a predefined motion or position;
adding or altering a virtual asset in a 3-D virtual environment in response to identifying the motion or position as the predefined motion or position;
moving or positioning a virtual representation of the first real-world performer based on the motion or position of the first real-world performer;
rendering a 2-D video stream of the virtual asset and the virtual representation of the first real-world performer in the 3-D virtual environment; and
causing a real-time video stream to be visible on a display device of a handheld or wearable device, wherein:
the handheld or wearable device captures a real-time view of a second real-world environment comprising a second real-world performer;
the real-time video stream comprises the 2-D video stream of the virtual asset and the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; and
the motion or position of the first real-world performer in the first real-world environment adds or alters the virtual asset in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with the virtual asset when viewed on the display device of the handheld or wearable device.

### Claim 9 (depends on 8)

The immersive content presentation system of claim 8, wherein the predefined motion or position comprises a hand gesture performed by the first real-world performer in the first real-world environment.

### Claim 10 (depends on 8)

The immersive content presentation system of claim 8, wherein the predefined motion or position comprises a location of the first real-world performer in the first real-world environment.

### Claim 11 (depends on 8)

The immersive content presentation system of claim 8, wherein identifying the motion or position as the predefined motion or position comprises:
receiving a motion-capture frame at a game engine of the immersive content presentation system; and
comparing the motion-capture frame to a plurality of motions or positions in a predefined library of motions or positions.

### Claim 12 (depends on 11)

The immersive content presentation system of claim 11, wherein:
the motion-capture frame comprises a plurality of vertices corresponding to visible markers on the first real-world performer in the first real-world environment; and
the immersive content presentation system generates motion vectors from the plurality of vertices that are compared to motion vectors in the predefined library of motions or positions.

### Claim 13 (depends on 8)

The immersive content presentation system of claim 8, wherein the 3-D virtual environment includes models or volumes that correspond to physical objects in the second real-world environment.

### Claim 14 (depends on 13)

The immersive content presentation system of claim 13, wherein rendering the 2-D video stream of the virtual asset in the 3-D virtual environment comprises:
rendering images in the 2-D video stream without images of the models or volumes that correspond to the physical objects in the second real-world environment.

### Claim 15 (depends on 13)

The immersive content presentation system of claim 13, wherein rendering the 2-D video stream of the virtual asset in the 3-D virtual environment comprises:
generating cutouts in images of the virtual asset in the 2-D video stream when the virtual asset is behind the models or volumes that correspond to the physical objects in the second real-world environment.

### Claim 16

A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a motion or position of a first real-world performer in a first real-world environment;
identifying the motion or position as a predefined motion or position;
adding or altering a virtual asset in a 3-D virtual environment in response to identifying the motion or position as the predefined motion or position;
moving or positioning a virtual representation of the first real-world performer based on the motion or position of the first real-world performer;
rendering a 2-D video stream of the virtual asset and the virtual representation of the first real-world performer in the 3-D virtual environment; and
causing a real-time video stream to be visible on a display device of a handheld or wearable device, wherein:
the handheld or wearable device captures a real-time view of a second real-world environment comprising a second real-world performer;
the real-time video stream comprises the 2-D video stream of the virtual asset and the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; and
the motion or position of the first real-world performer in the first real-world environment adds or alters the virtual asset in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with the virtual asset when viewed on the display device of the handheld or wearable device.

### Claim 17 (depends on 16)

The non-transitory, computer-readable medium of claim 16, wherein:
the display device comprises a pair of AR glasses; and
the 2-D video stream is composited with the real-time view of the second real-world environment by projecting the 2-D video stream on the pair of AR glasses in front of a natural view of the second real-world environment.

### Claim 18 (depends on 16)

The non-transitory, computer-readable medium of claim 16, wherein:
the display device comprises a tablet computer or mobile phone; and
the 2-D video stream is composited with the real-time view of the second real-world environment by compositing the 2-D video stream on the tablet computer or mobile phone in front of a view of the second real-world environment as captured by a camera of the tablet computer or mobile phone.

### Claim 19 (depends on 16)

The non-transitory, computer-readable medium of claim 16, wherein the second real-world environment comprises one or more human characters.

### Claim 20 (depends on 16)

The non-transitory, computer-readable medium of claim 16, wherein the second real-world environment comprises a plurality of depth cameras.