# US 2025/0148701 A1 · Dynamic overlapping of moving objects with real and virtual scenes for video see-through (vst) extended reality (xr)

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148701 A1
- Document ID: US20250148701A1
- Status: Published application
- Kind: A1
- Application number: 18/759,361
- Filed: 2024-06-28
- Published: 2025-05-08
- Earliest claimed: 2023-11-06
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Yingen Xiong
- Web page: https://patentgrep.com/patent/US20250148701A1

## CPC

- G06T 7/11: Region-based segmentation
- G06T 15/40: Hidden part removal
- G06T 7/13: Edge detection
- G06T 7/55: from multiple images
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20212: Image combination
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes obtaining image frames of a scene captured using one or more imaging sensors of a video see-through (VST) extended reality (XR) device and depth data associated with the scene. The image frames capture a moving object and static scene contents, and the moving object includes a portion of a body of a user. The method also includes generating masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene. The method further includes reconstructing images of the moving object and images of the static scene contents. In addition, the method includes combining the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images and rendering the combined images for presentation on at least one display of the VST XR device.

## Claims

### Claim 1

A method comprising:
obtaining image frames of a scene captured using one or more imaging sensors of a video see-through (VST) extended reality (XR) device and depth data associated with the scene, the image frames capturing a moving object and static scene contents, the moving object comprising a portion of a body of a user;
generating masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene;
reconstructing images of the moving object based on the image frames, the depth data, and the masks;
reconstructing images of the static scene contents based on the image frames and the depth data;
combining the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images; and
rendering the combined images for presentation on at least one display of the VST XR device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating first depth maps associated with the moving object based on the depth data; and
generating second depth maps associated with the static scene contents based on the depth data;
wherein the masks are generated based on the first depth maps;
wherein the images of the moving object are reconstructed based on the image frames, the first depth maps, and the masks; and
wherein the images of the static scene contents are reconstructed based on the image frames and the second depth maps.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the second depth maps comprises:
accessing a previous depth map corresponding to a previous image frame; and
for a current image frame, determining, based on the previous depth map, a depth for each portion of the scene behind the moving object that is disoccluded when the moving object moves.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
correcting for parallax associated with the moving object in the images of the moving object; and
separately correcting for parallax associated with the static scene contents in the images of the static scene contents.

### Claim 5 (depends on 1)

The method of claim 1, wherein combining the images of the moving object, the images of the static scene contents, and the one or more virtual features comprises:
overlapping the images of the moving object with the images of the static scene contents based on estimated boundaries of the moving object within the scene.

### Claim 6 (depends on 5)

The method of claim 5, wherein combining the images of the moving object, the images of the static scene contents, and the one or more virtual features further comprises:
performing hole filling to generate image content for each portion of the scene behind the moving object that is disoccluded when the moving object moves.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
estimating a latency associated with a pipeline of the VST XR device; and
modifying the images of the moving object and the images of the static scene contents or the combined images based on an estimated head pose of the user when the rendered combined images will be presented to the user.

### Claim 8

A video see-through (VST) extended reality (XR) device comprising:
one or more imaging sensors;
at least one display; and
at least one processing device configured to:
obtain image frames of a scene captured using the one or more imaging sensors and depth data associated with the scene, the image frames capturing a moving object and static scene contents, the moving object comprising a portion of a body of a user;
generate masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene;
reconstruct images of the moving object based on the image frames, the depth data, and the masks;
reconstruct images of the static scene contents based on the image frames and the depth data;
combine the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images; and
render the combined images for presentation on the at least one display.

### Claim 9 (depends on 8)

The VST XR device of claim 8, wherein:
the at least one processing device is further configured to:
generate first depth maps associated with the moving object based on the depth data; and
generate second depth maps associated with the static scene contents based on the depth data;
the at least one processing device is configured to generate the masks based on the first depth maps;
the at least one processing device is configured to reconstruct the images of the moving object based on the image frames, the first depth maps, and the masks; and
the at least one processing device is configured to reconstruct the images of the static scene contents based on the image frames and the second depth maps.

### Claim 10 (depends on 9)

The VST XR device of claim 9, wherein, to generate the second depth maps, the at least one processing device is configured to:
access a previous depth map corresponding to a previous image frame; and
for a current image frame, determine, based on the previous depth map, a depth for each portion of the scene behind the moving object that is disoccluded when the moving object moves.

### Claim 11 (depends on 8)

The VST XR device of claim 8, wherein the at least one processing device is further configured to:
correct for parallax associated with the moving object in the images of the moving object; and
separately correct for parallax associated with the static scene contents in the images of the static scene contents.

### Claim 12 (depends on 8)

The VST XR device of claim 8, wherein, to combine the images of the moving object, the images of the static scene contents, and the one or more virtual features, the at least one processing device is configured to overlap the images of the moving object with the images of the static scene contents based on estimated boundaries of the moving object within the scene.

### Claim 13 (depends on 12)

The VST XR device of claim 12, wherein, to combine the images of the moving object, the images of the static scene contents, and the one or more virtual features, the at least one processing device is further configured to perform hole filling to generate image content for each portion of the scene behind the moving object that is disoccluded when the moving object moves.

### Claim 14 (depends on 8)

The VST XR device of claim 8, wherein the at least one processing device is further configured to:
estimate a latency associated with a pipeline of the VST XR device; and
modify the images of the moving object and the images of the static scene contents or the combined images based on an estimated head pose of the user when the rendered combined images will be presented to the user.

### Claim 15

A non-transitory machine readable medium containing instructions that when executed cause at least one processor of a video see-through (VST) extended reality (XR) device to:
obtain image frames of a scene captured using one or more imaging sensors of the VST XR device and depth data associated with the scene, the image frames capturing a moving object and static scene contents, the moving object comprising a portion of a body of a user;
generate masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene;
reconstruct images of the moving object based on the image frames, the depth data, and the masks;
reconstruct images of the static scene contents based on the image frames and the depth data;
combine the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images; and
render the combined images for presentation on at least one display of the VST XR device.

### Claim 16 (depends on 15)

The non-transitory machine readable medium of claim 15, further containing instructions that when executed cause the at least one processor to:
generate first depth maps associated with the moving object based on the depth data; and
generate second depth maps associated with the static scene contents based on the depth data;
wherein the instructions when executed cause the at least one processor to:
generate the masks based on the first depth maps;
reconstruct the images of the moving object based on the image frames, the first depth maps, and the masks; and
reconstruct the images of the static scene contents based on the image frames and the second depth maps.

### Claim 17 (depends on 16)

The non-transitory machine readable medium of claim 16, wherein the instructions that when executed cause the at least one processor to generate the second depth maps comprise instructions that when executed cause the at least one processor to:
access a previous depth map corresponding to a previous image frame; and
for a current image frame, determine, based on the previous depth map, a depth for each portion of the scene behind the moving object that is disoccluded when the moving object moves.

### Claim 18 (depends on 15)

The non-transitory machine readable medium of claim 15, further containing instructions that when executed cause the at least one processor to:
correct for parallax associated with the moving object in the images of the moving object; and
separately correct for parallax associated with the static scene contents in the images of the static scene contents.

### Claim 19 (depends on 15)

The non-transitory machine readable medium of claim 15, wherein the instructions that when executed cause the at least one processor to combine the images of the moving object, the images of the static scene contents, and the one or more virtual features comprise instructions that when executed cause the at least one processor to:
overlap the images of the moving object with the images of the static scene contents based on estimated boundaries of the moving object within the scene; and
perform hole filling to generate image content for each portion of the scene behind the moving object that is disoccluded when the moving object moves.

### Claim 20 (depends on 15)

The non-transitory machine readable medium of claim 15, further containing instructions that when executed cause the at least one processor to:
estimate a latency associated with a pipeline of the VST XR device; and
modify the images of the moving object and the images of the static scene contents or the combined images based on an estimated head pose of the user when the rendered combined images will be presented to the user.