# US 12,217,365 B1 · Multiplexing video streams in an aggregate stream for a three-dimensional virtual environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,365 B1
- Document ID: US12217365B1
- Status: Granted
- Kind: B1
- Application number: 18/228,418
- Filed: 2023-07-31
- Published: 2025-02-06
- Granted: 2025-02-04
- Earliest claimed: 2023-07-31
- Assignees: Katmai Tech Inc.
- Applicants: Katmai Tech Inc.
- Inventors: Petr Polyakov
- Web page: https://patentgrep.com/patent/US12217365B1

## CPC

- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 15/04: Texture mapping
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/156: Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
- H04N 19/172: the region being a picture, frame or field
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In an aspect, a computer-implemented method allows for navigation in a three-dimensional (3D) virtual environment. In the method, data specifying a three-dimensional virtual space is received. A position and orientation in the three-dimensional virtual space is received. The position and orientation input by a first user and representing a first virtual camera used to render the three-dimensional virtual space to the first user. A video stream captured from a camera positioned to capture the first user is received. A second virtual camera is navigated according to an input of a second user.

## Claims

### Claim 1

A method for transferring video for generating avatars in a three-dimensional virtual environment, comprising:
(a) receiving a first video stream captured from a first camera on a first device of a first user, the first camera positioned to capture photographic images of the first user;
(b) receiving a second video stream captured from a second camera on a second device of a second user, the second camera positioned to capture photographic images of the second user;
(c) generating a third video stream incorporating the first and second video streams and metadata specifying how the first and second video streams are incorporated into the third video stream, wherein the generating comprises:
identifying a first area of frames from the third video stream;
identifying a second area of the frames from the third video stream;
for the respective frames of the third video stream:
identifying a first current frame from the first video stream;
identifying a second current frame from the second video stream;
mapping the first current frame onto the first area of the respective frame; and
mapping the second current frame onto the second area of the respective frame, wherein the metadata specifies the first and second areas and the mapping of the first and second areas to the respective first and second frames; and
(d) transmitting the third video stream and the metadata to a third device of a third user such that the third device is configured to:
(i) decode the first and second video streams from the third video stream based on the metadata,
(ii) generate, based on the decoded first video stream, a first avatar in the three-dimensional virtual environment at a first position of the first avatar controlled by the first user,
(iii) generate, based on the decoded second video stream, a second avatar in the three-dimensional virtual environment at a second position of the second avatar controlled by the second user, and
(iv) render, for display to the third user and from a perspective of a virtual camera controlled by the third user, the three-dimensional virtual environment including the first and second avatars.

### Claim 2 (depends on 1)

The method of claim 1, wherein the mapping the first current frame comprises resizing the first video stream to fit the first area.

### Claim 3 (depends on 2)

The method of claim 2, wherein the resizing comprises changing a resolution of the first video stream.

### Claim 4 (depends on 1)

The method of claim 1, wherein the mapping the first current frame comprises resizing the first video stream,
and wherein the mapping the second current frame comprises resizing the second video stream such that the first video stream is a different size than the second video stream.

### Claim 5 (depends on 4)

The method of claim 4, further comprising determining a relative size for the resized first and second video streams based on a priority value corresponding to a subscription level for the users of the respective first and second devices.

### Claim 6 (depends on 4)

The method of claim 4, further comprising determining a relative size for the resized first and second video streams based on whether an audio stream corresponding to the respective first and second video streams is muted.

### Claim 7 (depends on 4)

The method of claim 4, further comprising determining a relative size for the resized first and second video streams based on whether the virtual camera is located in the same audio zone as the first and second avatars.

### Claim 8 (depends on 4)

The method of claim 4, further comprising determining a relative size for the resized first and second video streams based on a relative distance of the first and second avatars from the virtual camera.

### Claim 9 (depends on 4)

The method of claim 4, further comprising determining a relative size for the resized first and second video streams based on how long the first and second video streams have been consumed by the third device.

### Claim 10 (depends on 4)

The method of claim 4, further comprising determining a relative size for the resized first and second video streams based on a relative quality of the first and second video streams.

### Claim 11 (depends on 1)

The method of claim 1, wherein the third device is configured to generate the first avatar (ii) by texture mapping respective frames of the first video stream onto a first three-dimensional model at the first position and is configured to generate the second avatar (iii) by texture mapping respective frames of the second video stream onto a second three-dimensional model at the second position.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
(e) providing to the third device a web application executable by a web browser in the third device, the web application including instructions that, when executed in the web browser, cause the third device to perform steps (i)-(iv).

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
(f) generating metadata describing how the first and second video streams are incorporated into the third video stream; and
(g) transmitting the metadata to the third device such that the third device is configured to decode the first and second video streams (i) based on the metadata.

### Claim 14 (depends on 1)

The method of claim 1, wherein the first and second areas are specified using a UV mapping.

### Claim 15 (depends on 1)

The method of claim 1, wherein the first and second areas are specified using steganography.

### Claim 16

A non-transitory computer readable medium including instructions for transferring video for generating avatars in a three-dimensional virtual environment that, when executed by a computing system, cause the computing system to perform operations comprising:
(a) receiving a first video stream captured from a first camera on a first device of a first user, the first camera positioned to capture photographic images of the first user;
(b) receiving a second video stream captured from a second camera on a second device of a second user, the second camera positioned to capture photographic images of the second user;
(c) generating a third video stream incorporating the first and second video streams and metadata specifying how the first and second video streams are incorporated into the third video stream, wherein the generating comprises:
identifying a first area of frames from the third video stream;
identifying a second area of the frames from the third video stream;
for the respective frames of the third video stream:
identifying a first current frame from the first video stream;
identifying a second current frame from the second video stream;
mapping the first current frame onto the first area of the respective frame; and
mapping the second current frame onto the second area of the respective frame, wherein the metadata specifies the first and second areas and the mapping of the first and second areas to the respective first and second frames; and
(d) transmitting the third video stream and the metadata to a third device of a third user such that the third device is configured to:
(i) decode the first and second video streams from the third video stream based on the metadata,
(ii) generate, based on the first decoded video stream, a first avatar in the three-dimensional virtual environment at a first position controlled by the first user,
(iii) generate, based on the second decoded video stream, a second avatar in the three-dimensional virtual environment at a second position controlled by the second user, and
(iv) render, for display to the third user and from a perspective of a virtual camera controlled by the third user, the three-dimensional virtual environment including the first and second avatars.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the mapping the first current frame comprises resizing the first video stream to fit the first area.

### Claim 18

A method for generating a three-dimensional virtual environment using a multiplexed video stream, the method comprising:
(a) receiving an aggregated video stream incorporating a first video stream captured from a first camera on a first device of a first user, the first camera positioned to capture photographic images of the first user, and a second video stream captured from a second camera on a second device of a second user, the second camera positioned to capture photographic images of the second user and metadata specifying how the first and second video streams are incorporated into the aggregated video stream;
(b) decoding the first and second video streams from the aggregated video stream based on the metadata;
(c) generating, based on the first decoded video stream, a first avatar in the three-dimensional virtual environment at a first position controlled by the first user;
(d) generating, based on the second decoded video stream, a second avatar in the three-dimensional virtual environment at a second position controlled by the second user; and
(e) rendering, for display to a third user and from a perspective of a virtual camera controlled by the third user, the three-dimensional virtual environment including the first and second avatars.
