Multiplexing video streams in an aggregate stream for a three-dimensional virtual environment

A non-transitory computer readable medium including instructions for transferring video for generating avatars in a three-dimensional virtual environment that, when executed by a computing system, cause the computing system to perform operations comprising:

• (a) receiving a first video stream captured from a first camera on a first device of a first user, the first camera positioned to capture photographic images of the first user;
• (b) receiving a second video stream captured from a second camera on a second device of a second user, the second camera positioned to capture photographic images of the second user;
• (c) generating a third video stream incorporating the first and second video streams and metadata specifying how the first and second video streams are incorporated into the third video stream, wherein the generating comprises:
• identifying a first area of frames from the third video stream;
• identifying a second area of the frames from the third video stream;
• for the respective frames of the third video stream:
• identifying a first current frame from the first video stream;
• identifying a second current frame from the second video stream;
• mapping the first current frame onto the first area of the respective frame;
• and
• mapping the second current frame onto the second area of the respective frame, wherein the metadata specifies the first and second areas and the mapping of the first and second areas to the respective first and second frames;
• and
• (d) transmitting the third video stream and the metadata to a third device of a third user such that the third device is configured to:
• (i) decode the first and second video streams from the third video stream based on the metadata,
• (ii) generate, based on the first decoded video stream, a first avatar in the three-dimensional virtual environment at a first position controlled by the first user,
• (iii) generate, based on the second decoded video stream, a second avatar in the three-dimensional virtual environment at a second position controlled by the second user, and
• (iv) render, for display to the third user and from a perspective of a virtual camera controlled by the third user, the three-dimensional virtual environment including the first and second avatars.

A method for generating a three-dimensional virtual environment using a multiplexed video stream, the method comprising:

• (a) receiving an aggregated video stream incorporating a first video stream captured from a first camera on a first device of a first user, the first camera positioned to capture photographic images of the first user, and a second video stream captured from a second camera on a second device of a second user, the second camera positioned to capture photographic images of the second user and metadata specifying how the first and second video streams are incorporated into the aggregated video stream;
• (b) decoding the first and second video streams from the aggregated video stream based on the metadata;
• (c) generating, based on the first decoded video stream, a first avatar in the three-dimensional virtual environment at a first position controlled by the first user;
• (d) generating, based on the second decoded video stream, a second avatar in the three-dimensional virtual environment at a second position controlled by the second user;
• and
• (e) rendering, for display to a third user and from a perspective of a virtual camera controlled by the third user, the three-dimensional virtual environment including the first and second avatars.