Dynamic overlapping of moving objects with real and virtual scenes for video see-through (vst) extended reality (xr)

A video see-through (VST) extended reality (XR) device comprising:

• one or more imaging sensors;
• at least one display;
• and
• at least one processing device configured to:
• obtain image frames of a scene captured using the one or more imaging sensors and depth data associated with the scene, the image frames capturing a moving object and static scene contents, the moving object comprising a portion of a body of a user;
• generate masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene;
• reconstruct images of the moving object based on the image frames, the depth data, and the masks;
• reconstruct images of the static scene contents based on the image frames and the depth data;
• combine the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images;
• and
• render the combined images for presentation on the at least one display.

A non-transitory machine readable medium containing instructions that when executed cause at least one processor of a video see-through (VST) extended reality (XR) device to:

• obtain image frames of a scene captured using one or more imaging sensors of the VST XR device and depth data associated with the scene, the image frames capturing a moving object and static scene contents, the moving object comprising a portion of a body of a user;
• generate masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene;
• reconstruct images of the moving object based on the image frames, the depth data, and the masks;
• reconstruct images of the static scene contents based on the image frames and the depth data;
• combine the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images;
• and
• render the combined images for presentation on at least one display of the VST XR device.