US 12,099,327 B2 · Holographic calling for artificial reality

Abstract

A holographic calling system can capture and encode holographic data at a sender-side of a holographic calling pipeline and decode and present the holographic data as a 3D representation of a sender at a receiver-side of the holographic calling pipeline. The holographic calling pipeline can include stages to capture audio, color images, and depth images; densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; convert depth images into a 3D mesh; paint the 3D mesh with color data; perform torso disocclusion; perform face reconstruction; and perform audio synchronization. In various implementations, different of these stages can be performed sender-side or receiver side. The holographic calling pipeline also includes sender-side compression, transmission over a communication channel, and receiver-side decompression and hologram output.

Claims20

Dependent claims

A method for conducting a holographic call using a holographic call pipeline, the method comprising:

establishing a communication channel between a sending device and at least one receiving device;
capturing, at the sending device, color, depth, and audio data and using the color and depth data to generate one or more color images and one or more depths images;
generating one or more masks for the one or more color images and one or more depth images;
applying the one or more masks to the one or more color images and one or more depth images to obtain masked portions of the one or more color images and one or more depth images;
compressing the masked portions of the one or more color images and one or more depth images;
and
synchronizing and transmitting, over the communication channel, the one or more color images, one or more depth images, and the audio data;
wherein the receiving device, in response to the transmitting:
decompresses the compressed portions;
converts the portions of the one or more depth images into a 3D mesh;
paints the portions of the one or more color images onto the 3D mesh;
synchronizes the audio data with the painted 3D mesh;
performs torso disocclusion on the 3D mesh;
performs facial reconstruction on the 3D mesh;
and
outputs the painted 3D mesh as a hologram with synchronized audio.

13 more elements

A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for conducting a holographic call using a holographic call pipeline, the process comprising:

capturing, at a sending device, color, depth, and audio data and using the color and depth data to generate one or more color images and one or more depths images;
generating one or more masks for the one or more color images and one or more depth images;
applying the one or more masks to the one or more color images and one or more depth images to obtain masked portions of the one or more color images and one or more depth images;
compressing the masked portions of the one or more color images and one or more depth images;
and
transmitting, over a communication channel, the compressed portions and the audio data;
wherein a receiving device, in response to the transmitting:
decompresses the compressed portions;
converts the portions of the one or more depth images into a 3D mesh;
paints the portions of the one or more color images onto the 3D mesh;
and
outputs the painted 3D mesh as a hologram with synchronized audio.

9 more elements

A computing system for conducting a holographic call using a holographic call pipeline, the computing system comprising:

one or more processors;
and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
capturing, at a sending device, color, depth, and audio data and using the color and depth data to generate one or more color images and one or more depths images;
generating one or more masks for the one or more color images and one or more depth images;
applying the one or more masks to the one or more color images and one or more depth images to obtain masked portions of the one or more color images and one or more depth images;
transmitting versions of the masked portions and the audio data;
wherein the receiving device, in response to the transmitting:
converts the portions of the one or more depth images and the portions of the one or more color images into a painted 3D mesh;
and
outputs the painted 3D mesh as a hologram, synchronized with the audio data.

8 more elements

Dependencies are inferred from the claim text.

Holographic calling for artificial reality

Life of this patent

Holographic calling for artificial reality

Life of this patent

Abstract

Claims20