# US 2023/0410414 A1 · Method and system for video transformation for video see-through augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410414 A1
- Document ID: US20230410414A1
- Status: Published application
- Kind: A1
- Application number: 18/052,827
- Filed: 2022-11-04
- Published: 2023-12-21
- Earliest claimed: 2022-06-16
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Yingen Xiong; Christopher A. Peri
- Web page: https://patentgrep.com/patent/US20230410414A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 15/20: Perspective computation
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/55: from multiple images
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/128: Adjusting depth or disparity
- H04N 13/246: Calibration of cameras
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 2207/10016: Video; Image sequence
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of video transformation for a video see-through (VST) augmented reality (AR) device includes obtaining video frames from multiple cameras associated with the VST AR device, where each video frame is associated with position data. The method also includes generating camera viewpoint depth maps associated with the video frames based on the video frames and the position data. The method further includes performing depth re-projection to transform the video frames from camera viewpoints to rendering viewpoints using the camera viewpoint depth maps. The method also includes performing hole filling of one or more holes created in one or more occlusion areas of at least one of the transformed video frames during the depth re-projection to generate at least one hole-filled video frame. In addition, the method includes displaying the transformed video frames including the at least one hole-filled video frame on multiple displays associated with the VST AR device.

## Claims

### Claim 1

A method of video transformation for a video see-through (VST) augmented reality (AR) device, the method comprising:
obtaining video frames from multiple cameras associated with the VST AR device, wherein each video frame is associated with position data;
generating camera viewpoint depth maps associated with the video frames based on the video frames and the position data;
performing depth re-projection to transform the video frames from camera viewpoints to rendering viewpoints using the camera viewpoint depth maps;
performing hole filling of one or more holes created in one or more occlusion areas of at least one of the transformed video frames during the depth re-projection to generate at least one hole-filled video frame; and
displaying the transformed video frames including the at least one hole-filled video frame on multiple displays associated with the VST AR device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining one or more hole areas for the one or more holes, the one or more hole areas corresponding to one or more areas of missing image data in the at least one of the transformed video frames.

### Claim 3 (depends on 2)

The method of claim 2, wherein performing the hole filling comprises, for each determined hole area:
determining whether a previous frame includes a pixel patch corresponding to the determined hole area; and
if it is determined that the previous frame includes the pixel patch corresponding to the determined hole area, filling the determined hole area with the pixel patch.

### Claim 4 (depends on 3)

The method of claim 3, wherein performing the hole filling further comprises, for each determined hole area:
if it is determined that the previous frame does not include the pixel patch corresponding to the determined hole area, filling the determined hole area using pixel expansion.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
detecting positions of eyes of a user viewing the VST AR device relative to the displays of the VST AR device;
wherein the rendering viewpoints are based on the detected positions of the eyes of the user.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining rendering viewpoint depth maps corresponding to the transformed video frames based on the detected positions of the eyes of the user.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
detecting the one or more holes created in the one or more occlusion areas of the at least one of the transformed video frames based on the rendering viewpoint depth maps.

### Claim 8

A video see-through (VST) augmented reality (AR) device comprising:
multiple cameras; and
at least one processing device configured to:
obtain video frames from the cameras, wherein each video frame is associated with position data;
generate camera viewpoint depth maps associated with the video frames based on the video frames and the position data;
perform depth re-projection to transform the video frames from camera viewpoints to rendering viewpoints using the camera viewpoint depth maps;
perform hole filling of one or more holes created in one or more occlusion areas of at least one of the transformed video frames during the depth re-projection to generate at least one hole-filled video frame; and
initiate display of the transformed video frames including the at least one hole-filled video frame on multiple displays associated with the VST AR device.

### Claim 9 (depends on 8)

The VST AR device of claim 8, wherein the at least one processing device is further configured to determine one or more hole areas for the one or more holes, the one or more hole areas corresponding to one or more areas of missing image data in the at least one of the transformed video frames.

### Claim 10 (depends on 9)

The VST AR device of claim 9, wherein, to perform the hole filling, the at least one processing device is configured, for each determined hole area, to:
determine whether a previous frame includes a pixel patch corresponding to the determined hole area; and
if it is determined that the previous frame includes the pixel patch corresponding to the determined hole area, fill the determined hole area with the pixel patch.

### Claim 11 (depends on 10)

The VST AR device of claim 10, wherein, to perform the hole filling, the at least one processing device is further configured, for each determined hole area, to:
if it is determined that the previous frame does not include the pixel patch corresponding to the determined hole area, fill the determined hole area using pixel expansion.

### Claim 12 (depends on 8)

The VST AR device of claim 8, wherein:
the at least one processing device is further configured to detect positions of eyes of a user viewing the VST AR device relative to the displays of the VST AR device; and
the rendering viewpoints are based on the detected positions of the eyes of the user.

### Claim 13 (depends on 12)

The VST AR device of claim 12, wherein the at least one processing device is further configured to determine rendering viewpoint depth maps corresponding to the transformed video frames based on the detected positions of the eyes of the user.

### Claim 14 (depends on 13)

The VST AR device of claim 13, wherein the at least one processing device is further configured to detect the one or more holes created in the one or more occlusion areas of the at least one of the transformed video frames based on the rendering viewpoint depth maps.

### Claim 15

A non-transitory machine readable medium containing instructions that when executed cause at least one processor of a video see-through (VST) augmented reality (AR) device to:
obtain video frames from multiple cameras associated with the VST AR device, wherein each video frame is associated with position data;
generate camera viewpoint depth maps associated with the video frames based on the video frames and the position data;
perform depth re-projection to transform the video frames from camera viewpoints to rendering viewpoints using the camera viewpoint depth maps;
perform hole filling of one or more holes created in one or more occlusion areas of at least one of the transformed video frames during the depth re-projection to generate at least one hole-filled video frame; and
initiate display of the transformed video frames including the at least one hole-filled video frame on multiple displays associated with the VST AR device.

### Claim 16 (depends on 15)

The non-transitory machine readable medium of claim 15, further containing instructions that when executed cause the at least one processor to determine one or more hole areas for the one or more holes, the one or more hole areas corresponding to one or more areas of missing image data in the at least one of the transformed video frames.

### Claim 17 (depends on 16)

The non-transitory machine readable medium of claim 16, wherein the instructions that when executed cause the at least one processor to perform the hole filling comprise instructions that when executed cause the at least one processor, for each determined hole area, to:
determine whether a previous frame includes a pixel patch corresponding to the determined hole area; and
if it is determined that the previous frame includes the pixel patch corresponding to the determined hole area, fill the determined hole area with the pixel patch.

### Claim 18 (depends on 17)

The non-transitory machine readable medium of claim 17, wherein the instructions that when executed cause the at least one processor to perform the hole filling further comprise instructions that when executed cause the at least one processor, for each determined hole area, to:
if it is determined that the previous frame does not include the pixel patch corresponding to the determined hole area, fill the determined hole area using pixel expansion.

### Claim 19 (depends on 15)

The non-transitory machine readable medium of claim 15, further containing instructions that when executed cause the at least one processor to detect positions of eyes of a user viewing the VST AR device relative to the displays of the VST AR device;
wherein the rendering viewpoints are based on the detected positions of the eyes of the user.

### Claim 20 (depends on 19)

The non-transitory machine readable medium of claim 19, further containing instructions that when executed cause the at least one processor to:
determine rendering viewpoint depth maps corresponding to the transformed video frames based on the detected positions of the eyes of the user; and
detect the one or more holes created in the one or more occlusion areas of the at least one of the transformed video frames based on the rendering viewpoint depth maps.
