# US 12,603,978 B2 · System and method for parallax correction for video see-through augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,603,978 B2
- Document ID: US12603978B2
- Status: Granted
- Kind: B2
- Application number: 18/302,622
- Filed: 2023-04-18
- Published: 2024-04-11
- Granted: 2026-04-14
- Earliest claimed: 2022-09-30
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Yingen Xiong
- Web page: https://patentgrep.com/patent/US12603978B2

## CPC

- H04N 13/128: Adjusting depth or disparity
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/593: from stereo images
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 2207/10024: Color image
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes obtaining a stereo image pair including a first image and a second image. The method also includes generating a first feature map of the first image and a second feature map of the second image, the first and second feature maps including extracted positions associated with pixels in the images. The method further includes generating a disparity map between the first and second images based on a dense depth map. The method also includes generating a verified depth map based on a pixelwise comparison of predicted positions and the extracted positions associated with at least some of the pixels in at least one of the images, the predicted positions determined based on the disparity map. In addition, the method includes generating a first virtual view and a second virtual view to present on a display panel of an VST AR device based on the verified depth map.

## Claims

### Claim 1

A method comprising:
obtaining a stereo image pair including a first image and a second image captured using first and second see-through cameras associated with a video see-through (VST) augmented reality (AR) device;
generating a first feature map of the first image and a second feature map of the second image, the first feature map including extracted positions associated with pixels in the first image, the second feature map including extracted positions associated with pixels in the second image;
generating a disparity map between the first and second images based on a dense depth map;
generating a verified depth map based on a pixelwise comparison of predicted positions and the extracted positions associated with at least some of the pixels in at least one of the first and second images, the predicted positions determined based on the disparity map, and wherein, for each pixel in the first image that represents a point in space, either a depth value or a corrected depth value is stored in the verified depth map based on a threshold; and
generating a first virtual view and a second virtual view to present on a display panel of the VST AR device based on the verified depth map, wherein generating the first virtual view and the second virtual view based on the verified depth map comprises warping the verified depth map to generate a first warped depth map for a first virtual camera and a second warped depth map for a second virtual camera.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the verified depth map comprises:
for each pixel in the first image that represents a point in space:
determining a predicted position of a pixel in the second image representing the point based on the extracted position of that pixel in the first feature map and a disparity value associated with that pixel from the disparity map;
comparing the predicted position of the pixel in the second image to the extracted position of the pixel in the second feature map; and
one of:
in response to a difference between the predicted position and the extracted position being less than the threshold, storing, in the verified depth map, the depth value associated with that pixel in the dense depth map; and
in response to the difference between the predicted position and the extracted position being greater than the threshold, determining the corrected depth value and storing the corrected depth value in the verified depth map.

### Claim 3 (depends on 2)

The method of claim 2, wherein the predicted position of the pixel in the second image is determined using an equation:
x
r
=
x
l
-
Bf
d
where xr represents the predicted position of the pixel in the second image, xl represents the extracted position of the pixel in the first feature map, B represents a distance between first and second virtual camera positions associated with the VST AR device, f represents a focal length, and d represents a depth value associated with the pixel.

### Claim 4 (depends on 2)

The method of claim 2, wherein the corrected depth value is determined based on the dense depth map and color and spatial information from at least one of the first and second images.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the first virtual view and the second virtual view based on the verified depth map further comprises:
generating first and second virtual objects using the first and second warped depth maps;
generating the first virtual view using the first warped depth map and real world information; and
generating the second virtual view using the second warped depth map and the real world information.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating the dense depth map using a deep neural network (DNN).

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
presenting the first virtual view and the second virtual view on the display panel of the VST AR device.

### Claim 8

A video see-through (VST) augmented reality (AR) device comprising:
at least one display panel;
first and second see-through cameras; and
at least one processing device configured to:
obtain a stereo image pair including a first image and a second image captured using the first and second see-through cameras;
generate a first feature map of the first image and a second feature map of the second image, the first feature map including extracted positions associated with pixels in the first image, the second feature map including extracted positions associated with pixels in the second image;
generate a disparity map between the first and second images based on a dense depth map;
generate a verified depth map based on a pixelwise comparison of predicted positions and the extracted positions associated with at least some of the pixels in at least one of the first and second images, the predicted positions determined based on the disparity map, and wherein, for each pixel in the first image that represents a point in space, either a depth value or a corrected depth value is stored in the verified depth map based on a threshold; and
generate a first virtual view and a second virtual view to present on a display panel of the VST AR device based on the verified depth map, wherein to generate the first virtual view and the second virtual view based on the verified depth map, the at least one processing device is configured to warp the verified depth map to generate a first warped depth map for a first virtual camera and a second warped depth map for a second virtual camera.

### Claim 9 (depends on 8)

The VST AR device of claim 8, wherein to generate the verified depth map, the at least one processing device is configured to:
for each pixel in the first image that represents a point in space:
determine a predicted position of a pixel in the second image representing the point based on the extracted position of that pixel in the first feature map and a disparity value associated with that pixel from the disparity map;
compare the predicted position of the pixel in the second image to the extracted position of the pixel in the second feature map; and
one of:
in response to a difference between the predicted position and the extracted position being less than the threshold, store, in the verified depth map, the depth value associated with that pixel in the dense depth map; and
in response to the difference between the predicted position and the extracted position being greater than the threshold, determine the corrected depth value and store the corrected depth value in the verified depth map.

### Claim 10 (depends on 9)

The VST AR device of claim 9, wherein the at least one processing device is configured to determine the predicted position of the pixel in the second image using an equation:
x
r
=
x
l
-
Bf
d
where xr represents the predicted position of the pixel in the second image, xl represents the extracted position of the pixel in the first image feature map, B represents a distance between first and second virtual camera positions associated with the VST AR device, f represents a focal length, and d represents a depth value associated with the pixel.

### Claim 11 (depends on 9)

The VST AR device of claim 9, wherein the at least one processing device is configured to determine the corrected depth value based on the dense depth map and color and spatial information from at least one of the first and second images.

### Claim 12 (depends on 8)

The VST AR device of claim 8, wherein to generate the first virtual view and the second virtual view based on the verified depth map, the at least one processing device is further configured to:
generate first and second virtual objects using the first and second warped depth maps;
generate the first virtual view using the first warped depth map and real world information; and
generate the second virtual view using the second warped depth map and the real world information.

### Claim 13 (depends on 8)

The VST AR device of claim 8, wherein the at least one processing device is further configured to:
generate the dense depth map using a deep neural network (DNN).

### Claim 14 (depends on 8)

The VST AR device of claim 8, wherein the at least one processing device is further configured to:
control the display panel to present the first virtual view and the second virtual view.

### Claim 15

A non-transitory machine readable medium containing instructions that when executed cause at least one processor to:
obtain a stereo image pair including a first image and a second image captured using first and second see-through cameras associated with a video see-through (VST) augmented reality (AR) device;
generate a first feature map of the first image and a second feature map of the second image, the first feature map including extracted positions associated with pixels in the first image, the second feature map including extracted positions associated with pixels in the second image;
generate a disparity map between the first and second images based on a dense depth map;
generate a verified depth map based on a pixelwise comparison of predicted positions and the extracted positions associated with at least some of the pixels in at least one of the first and second images, the predicted positions determined based on the disparity map, and wherein, for each pixel in the first image that represents a point in space, either a depth value or a corrected depth value is stored in the verified depth map based on a threshold; and
generate a first virtual view and a second virtual view to present on a display panel of the VST AR device based on the verified depth map, wherein the instructions that cause the at least one processor to generate the first virtual view and the second virtual view based on the verified depth map comprise instructions to warp the verified depth map to generate a first warped depth map for a first virtual camera and a second warped depth map for a second virtual camera.

### Claim 16 (depends on 15)

The non-transitory machine readable medium of claim 15, wherein the instructions that cause the at least one processor to generate the verified depth map comprise instructions to:
for each pixel in the first image that represents a point in space:
determine a predicted position of a pixel in the second image representing the point based on the extracted position of that pixel in the first feature map and a disparity value associated with that pixel from the disparity map;
compare the predicted position of the pixel in the second image to the extracted position of the pixel in the second feature map; and
one of:
in response to a difference between the predicted position and the extracted position being less than the threshold, store, in the verified depth map, the depth value associated with that pixel in the dense depth map; and
in response to the difference between the predicted position and the extracted position being greater than the threshold, determine the corrected depth value and store the corrected depth value in the verified depth map.

### Claim 17 (depends on 16)

The non-transitory machine readable medium of claim 16, wherein the predicted position of the pixel in the second image is determined using an equation:
x
r
=
x
l
-
Bf
d
where xr represents the predicted position of the pixel in the second image, xl represents the extracted position of the pixel in the first feature map, B represents a distance between first and second virtual camera positions associated with the VST AR device, f represents a focal length, and d represents a depth value associated with the pixel.

### Claim 18 (depends on 16)

The non-transitory machine readable medium of claim 16, wherein the corrected depth value is determined based on the dense depth map and color and spatial information from at least one of the first and second images.

### Claim 19 (depends on 15)

The non-transitory machine readable medium of claim 15, wherein the instructions that cause the at least one processor to generate the first virtual view and the second virtual view based on the verified depth map further comprise instructions to:
generate first and second virtual objects using the first and second warped depth maps;
generate the first virtual view using the first warped depth map and real world information; and
generate the second virtual view using the second warped depth map and the real world information.

### Claim 20 (depends on 15)

The non-transitory machine readable medium of claim 15, further containing instructions that when executed cause the at least one processor to:
generate the dense depth map using a deep neural network (DNN).