Systems and methods for spatially-anchored volumetric telepresence and clinical image visualization

A method for volumetric telepresence, comprising:

• acquiring, with a plurality of depth-sensing cameras positioned around a capture zone, synchronized depth and color data of a participant in real time;
• reconstructing, by a processing subsystem, a volumetric representation of the participant from the synchronized depth and color data;
• encoding and transmitting, by the processing subsystem, the volumetric representation as a data stream over a network to a remote spatial display device;
• receiving, by the spatial display device, the transmitted volumetric representation;
• decoding, by the spatial display device, the volumetric representation;
• rendering, by the spatial display device, a hologram of the participant spatially anchored within a local physical environment of the spatial display device;
• and
• maintaining spatial registration between a coordinate system of the local physical environment and the capture zone such that the hologram appears at a corresponding position and orientation within the capture zone and the local physical environment,
• wherein the spatial registration is dynamically updated during motion of the participant, and the volumetric representation is maintained in real-time synchronization for conversational interaction.

A system for headset-aware volumetric telepresence, comprising:

• one or more depth-sensing cameras positioned around a capture zone and configured to acquire synchronized depth and color data of a participant in real time;
• a processing subsystem configured to reconstruct a volumetric representation of the participant from the synchronized depth and color data;
• and
• a headset worn by the participant and including one or more inward-facing sensors configured to capture facial-expression data of the participant while the participant's face is partially occluded by the headset;
• the processing subsystem being further configured to:
• generate, during a pre-conference enrollment phase, a personalized three-dimensional facial model of the participant based on multi-view depth and color imagery of the participant's uncovered face;
• receive the facial-expression data from the headset;
• animate the personalized facial model according to the facial-expression data;
• merge the animated facial model with the reconstructed volumetric representation of the participant to produce a composite volumetric representation having a reconstructed face region;
• and
• encode and transmit the composite volumetric representation to a remote spatial display device for real-time rendering as a holographic image including the participant's facial features.