# US 2023/0274455 A1 · Systems and methods for low compute high-resolution depth map generation using low-resolution cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274455 A1
- Document ID: US20230274455A1
- Status: Published application
- Kind: A1
- Application number: 18/295,768
- Filed: 2023-04-04
- Published: 2023-08-31
- Earliest claimed: 2021-04-20
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Raymond Kirk PRICE; Michael BLEYER; Christopher Douglas EDMONDS
- Web page: https://patentgrep.com/patent/US20230274455A1

## CPC

- G06T 7/593: from stereo images
- G02B 27/017: {Head mounted}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/20: using local operators
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/73: Deblurring; Sharpening
- G06T 2207/10012: Stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10048: Infrared image
- G06T 2207/20028: Bilateral filtering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A system for low compute high-resolution depth map generation using low-resolution cameras is configured to obtain a stereo pair of images and generate a depth map by performing stereo matching on the stereo pair of images. The system is also configured to obtain a first image comprising first texture information for the environment that has a first image resolution that is higher than an image resolution of images of the stereo pair of images. The system is further configured to generate a reprojected first image by reprojecting the first image to correspond to an image capture perspective associated with the depth map. The reprojection of the first image is based on depth information from the depth map and includes reprojected first texture information for the environment. The system is also configured to generate an upsampled depth map based on the depth map.

## Claims

### Claim 1

A system for facilitating low compute depth map generation to provide parallax-corrected images, the system comprising:
one or more processors; and
one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate low compute depth map generation to provide parallax-corrected images by configuring the system to:
obtain a first image of an environment;
obtain a second image of the environment, the second image capturing the environment in temporal synchronization with the first image, the second image comprising a higher image resolution than the first image;
generate an upsampled first image, wherein the upsampled first image comprises a same image resolution as the second image; and
generate a depth map of the environment by performing stereo matching on the upsampled first image and the second image.

### Claim 2 (depends on 1)

The system of claim 1, wherein the instructions are executable by the one or more processors to further configure the system to generate a parallax-corrected image by reprojecting the second image to correspond to a user perspective.

### Claim 3 (depends on 2)

The system of claim 2, wherein the reprojection of the second image for generating the parallax-corrected image is based on depth information from the depth map.

### Claim 4 (depends on 1)

The system of claim 1, wherein generating the upsampled first image comprises utilizing an edge-preserving filter.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions are executable by the one or more processors to further configure the system to generate a parallax-corrected image by reprojecting the reprojected first image to correspond to a user perspective, the reprojection of the reprojected first image for generating the parallax-corrected image being based on upsampled depth information from the upsampled depth map.

### Claim 6 (depends on 5)

The system of claim 5, wherein the system further comprises a display, and wherein the instructions are executable by the one or more processors to further configure the system to display the parallax-corrected image on the display.

### Claim 7 (depends on 1)

The system of claim 1, wherein the system comprises a stereo camera pair and a first camera.

### Claim 8 (depends on 7)

The system of claim 7, wherein the second image is captured by one camera of the stereo camera pair, and wherein the first image is captured by the first camera.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one camera of the stereo camera pair and the first camera comprise a same camera modality.

### Claim 10 (depends on 9)

The system of claim 9, wherein the first camera comprises a thermal camera.

### Claim 11 (depends on 9)

The system of claim 9, wherein an other camera of the stereo camera pair comprises a different camera modality than the first camera.

### Claim 12 (depends on 11)

The system of claim 11, wherein the other camera of the stereo camera pair comprises a low light camera.

### Claim 13 (depends on 1)

The system of claim 1, wherein the system comprises a head-mounted display (HMD).

### Claim 14

A system for facilitating low compute depth map generation to provide parallax-corrected images, the system comprising:
one or more processors; and
one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate low compute depth map generation to provide parallax-corrected images by configuring the system to:
obtain a first image of an environment;
obtain a second image comprising texture information for the environment, the second image comprising a higher image resolution than the first image;
generate a downsampled second image, wherein the downsampled second image comprises a same image resolution as the first image;
generate a depth map of the environment by performing stereo matching on the downsampled second image and the first image; and
generate an upsampled depth map based on the depth map and the texture information of the second image.

### Claim 15 (depends on 14)

The system of claim 14, wherein the first image and the second image are associated with a same camera modality, and wherein the instructions are executable by the one or more processors to further configure the system to:
generate a parallax-corrected image by reprojecting the second image to correspond to a user perspective, the reprojection of the second image for generating the parallax-corrected image being based on upsampled depth information from the upsampled depth map.

### Claim 16 (depends on 15)

The system of claim 15, wherein the instructions are executable by the one or more processors to further configure the system to:
obtain an additional image comprising additional texture information for the environment, the additional image comprising a higher image resolution than the first image, wherein the additional image is associated with a different camera modality than the first image and the second image;
generate a reprojected additional image by reprojecting the additional image to correspond to an image capture perspective associated with the upsampled depth map, the reprojected additional image comprising reprojected additional texture information for the environment; and
generate a parallax-corrected additional image by reprojecting the reprojected additional image to correspond to a user perspective.

### Claim 17 (depends on 16)

The system of claim 16, wherein the reprojection of the reprojected additional image for generating the parallax-corrected additional image is based on the upsampled depth information from the upsampled depth map.

### Claim 18

A method for facilitating low compute depth map generation to provide parallax-corrected images, the method comprising:
obtaining a first image of an environment;
obtaining a second image of the environment, the second image capturing the environment in temporal synchronization with the first image, the second image comprising a higher image resolution than the first image;
generating an upsampled first image, wherein the upsampled first image comprises a same image resolution as the second image; and
generating a depth map of the environment by performing stereo matching on the upsampled first image and the second image.

### Claim 19 (depends on 18)

The method of claim 18, further comprising generating a parallax-corrected image by reprojecting the second image to correspond to a user perspective.

### Claim 20 (depends on 19)

The method of claim 19, wherein the reprojection of the second image for generating the parallax-corrected image is based on depth information from the depth map.
