# US 2024/0177328 A1 · Continuous surface and depth estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177328 A1
- Document ID: US20240177328A1
- Status: Published application
- Kind: A1
- Application number: 18/435,797
- Filed: 2024-02-07
- Published: 2024-05-30
- Earliest claimed: 2021-05-18
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Olha Borys; Ilteris Kaan Canberk; Daniel Wagner; Jakob Zillner
- Web page: https://patentgrep.com/patent/US20240177328A1

## CPC

- G06T 7/593: from stereo images
- G06T 7/73: using feature-based methods
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/128: Adjusting depth or disparity
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- G06T 2207/10012: Stereo images
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed are systems, methods, and non-transitory computer-readable media for continuous surface and depth estimation. A continuous surface and depth estimation system determines the depth and surface normal of physical objects by using stereo vision limited within a predetermined window.

## Claims

### Claim 1

A method comprising:
detecting a number of matching features within a first image and a second image that depict an object;
determining the number of matching features is below a threshold value;
identifying a known surface plane responsive to the determining that the number of matching features is below the threshold value;
performing ray casting upon the known surface plane;
determining a depth value for the object based on the ray casting; and
causing presentation of media content based on the depth value.

### Claim 2 (depends on 1)

The method of claim 1, wherein the detecting the number of matching features within the first image and the second image includes:
detecting the set of matching features within a predetermined window of the first image and the second image, the set of matching features comprising a first feature depicted within the predetermined window of the first image, and a second feature depicted within the predetermined window of the second image.

### Claim 3 (depends on 2)

The method of claim 2, wherein the predetermined window corresponds with a sub-portion of the first image and the second image.

### Claim 4 (depends on 2)

The method of claim 2, wherein the predetermined window is a center portion of the first image and the second image.

### Claim 5 (depends on 1)

The method of claim 1, wherein the media content includes Augmented-Reality (AR) content.

### Claim 6 (depends on 1)

The method of claim 1, wherein the known surface plane includes a previous surface plane determined based on a set of matching features in a pair of images.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first image captured by a first optical sensor and the second image captured by a second optical sensor.

### Claim 8

A system comprising:
one or more computer processors; and
one or more computer readable mediums storing instructions that, when executed by the one or more computer processors, causes the system to perform operations comprising:
detecting a number of matching features within a first image and a second image that depict an object;
determining the number of matching features is below a threshold value;
identifying a known surface plane responsive to the determining that the number of matching features is below the threshold value;
performing ray casting upon the known surface plane;
determining a depth value for the object based on the ray casting; and
causing presentation of media content based on the depth value.

### Claim 9 (depends on 8)

The system of claim 8, wherein the detecting the number of matching features within the first image and the second image includes:
detecting the set of matching features within a predetermined window of the first image and the second image, the set of matching features comprising a first feature depicted within the predetermined window of the first image, and a second feature depicted within the predetermined window of the second image.

### Claim 10 (depends on 9)

The system of claim 9, wherein the predetermined window corresponds with a sub-portion of the first image and the second image.

### Claim 11 (depends on 9)

The system of claim 9, wherein the predetermined window is a center portion of the first image and the second image.

### Claim 12 (depends on 8)

The system of claim 8, wherein the media content includes Augmented-Reality (AR) content.

### Claim 13 (depends on 8)

The system of claim 8, wherein the known surface plane includes a previous surface plane determined based on a set of matching features in a pair of images.

### Claim 14 (depends on 8)

The system of claim 8, wherein the first image captured by a first optical sensor and the second image captured by a second optical sensor.

### Claim 15

A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more computing devices to perform operations comprising:
detecting a number of matching features within a first image and a second image that depict an object;
determining the number of matching features is below a threshold value;
identifying a known surface plane responsive to the determining that the number of matching features is below the threshold value;
performing ray casting upon the known surface plane;
determining a depth value for the object based on the ray casting; and
causing presentation of media content based on the depth value.

### Claim 16 (depends on 15)

The non-transitory machine-readable storage medium of claim 15, wherein the detecting the number of matching features within the first image and the second image includes:
detecting the set of matching features within a predetermined window of the first image and the second image, the set of matching features comprising a first feature depicted within the predetermined window of the first image, and a second feature depicted within the predetermined window of the second image.

### Claim 17 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, wherein the predetermined window corresponds with a sub-portion of the first image and the second image.

### Claim 18 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, wherein the predetermined window is a center portion of the first image and the second image.

### Claim 19 (depends on 15)

The non-transitory machine-readable storage medium of claim 15, wherein the media content includes Augmented-Reality (AR) content.

### Claim 20 (depends on 15)

The non-transitory machine-readable storage medium of claim 15, wherein the known surface plane includes a previous surface plane determined based on a set of matching features in a pair of images.
