Seamless injection of augmented three-dimensional imagery using a positionally encoded video stream

One or more non-transitory computer-readable storage media coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

• formulating a geometry of an environment that is in proximity to a host peer, based on sensor data collected from the environment by at least one sensor;
• determining, by the host peer, an occlusion mesh of the environment based on the formulated geometry;
• capturing, with a camera, a plurality of video frames of the environment and spatial transform data of the camera within the geometry of the environment for each captured video frame;
• synchronizing each of the video frames with the respective spatial transform data of the camera and the occlusion mesh;
• and
• communicating, to a first remote peer that is remote from the host peer, the video frames, the synchronized spatial transform data of the camera, and the occlusion mesh;
• wherein the first remote peer is configured to generate a local composite of a first 3D image overlay with each of the video frames such that the first 3D image overlay is synchronized to each of the video frames, wherein the first 3D image overlay is occluded by the occlusion mesh relative to a respective correspondence within the environment to generate the local composite that is oriented and occluded relative to each of the video frames and the synchronized spatial transform data, wherein the first 3D image overlay is generated on the first remote peer;
• and
• wherein the first remote peer is further configured to display the local composite as a 3D AR of the environment that is in proximity to the host peer.

A system, comprising:

• one or more remote peers each comprising a three-dimensional (3D) accelerated graphics chip and a display;
• a one or more processors;
• and
• a computer-readable storage device coupled to the one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
• formulating a geometry of an environment that is in proximity to a host peer, based on sensor data collected from the environment by at least one sensor;
• determining, by the host peer, an occlusion mesh of the environment based on the formulated geometry;
• capturing, with a camera, a plurality of video frames of the environment and spatial transform data of the camera within the geometry of the environment for each captured video frame;
• synchronizing each of the video frames with the respective spatial transform data of the camera and the occlusion mesh;
• and
• communicating, to a first remote peer that is remote from the host peer, the video frames, the synchronized spatial transform data of the camera, and the occlusion mesh;
• wherein the first remote peer is configured to generate a local composite of a first 3D image overlay with each of the video frames such that the first 3D image overlay is synchronized to each of the video frames, wherein the first 3D image overlay is occluded by the occlusion mesh relative to a respective correspondence within the environment to generate the local composite that is oriented and occluded relative to each of the video frames and the synchronized spatial transform data, wherein the first 3D image overlay is generated on the first remote peer;
• and
• wherein the first remote peer is further configured to display the local composite as a 3D AR of the environment that is in proximity to the host peer.