# US 2025/0157138 A1 · Three dimensional scene inpainting using stereo extraction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157138 A1
- Document ID: US20250157138A1
- Status: Published application
- Kind: A1
- Application number: 19/022,128
- Filed: 2025-01-15
- Published: 2025-05-15
- Earliest claimed: 2018-09-27
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Nir DAUBE; Zachi KARNI
- Web page: https://patentgrep.com/patent/US20250157138A1

## CPC

- G06T 7/593: from stereo images
- G06T 15/205: {Image-based rendering}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 7/55: from multiple images
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10012: Stereo images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for rendering three-dimensional (3D) scenes having improved visual characteristics from a pair of 2D images having different viewpoints. The 3D scene is created by obtaining a first two-dimensional (2D) image of a scene object from a first viewpoint, obtaining a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint, creating a depth map from the first and second 2D images, creating a 3D scene from the depth map and the first and second 2D images, detecting regions of the initial 3D scene with incomplete image information, reconstructing the detected regions of the 3D scene, determining replacement information and modify the reconstructed regions, and rendering the 3D scene with the modified reconstructed regions.

## Claims

### Claim 1

A system for creating a three-dimensional (3D) scene, the system comprising:
eyewear including a first imager and a second imager spaced from the first imager, the first imager configured to obtain a first two-dimensional (2D) image of a scene object from a first viewpoint and the second imager configured to obtain a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint; and
a processing system coupled to the eyewear, the processing system configured to:
create a 3D scene from the first and second 2D images;
detect regions of the 3D scene with incomplete image information by identifying contiguous faces of the 3D scene with a side that is less than a predefined amount of another side or that is below a threshold dimension;
reconstruct the detected regions of the 3D scene;
determine replacement information and modify the reconstructed regions; and
render the 3D scene with the modified reconstructed regions.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processing system is further configured to:
identify holes in the rendered 3D scene from one or more viewpoints; and
refine the rendered 3D scene to fill in the holes.

### Claim 3 (depends on 1)

The system of claim 1, wherein the eyewear includes a first temple and a second temple and wherein the first imager is adjacent the first temple and the second imager is adjacent the second temple.

### Claim 4 (depends on 1)

The system of claim 1, wherein to determine replacement information for the detected regions with incomplete image information the processing system is configured to:
identify a boundary surrounding each detected region;
identify background information in the boundary surrounding each detected region;
identify foreground information in the boundary surrounding each detected region; and
blend the background and foreground information through each detected region giving the background information higher weight than the foreground information.

### Claim 5 (depends on 1)

The system of claim 1, wherein the predefined amount is five percent.

### Claim 6 (depends on 1)

The system of claim 1, wherein the threshold dimension is one millimeter.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processing system is further configured to:
create a depth map including pixel vertices and corresponding image information from the first and second 2D images;
wherein to create the 3D scene the processing system connects the vertices to form first faces and to construct the detected regions the processing system connects the vertices of boundary regions to form second faces, the second faces being different than the first faces.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processing system is further configured to:
create a depth map including pixel vertices and confidence values corresponding to each vertex;
wherein to create the 3D scene the processing system is configured to connect the vertices to form the faces and to detect the regions of the 3D scene with incomplete information the processing system is further configured to identify contiguous faces including low confidence faces including at least one vertex generated with inconsistent values between the first 2D image and the second 2D image.

### Claim 9

A method for creating a three-dimensional (3D) scene, the method comprising:
obtaining a first two-dimensional (2D) image of a scene object from a first viewpoint;
obtaining a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint;
creating a 3D scene from the first and second 2D images;
detecting regions of the 3D scene with incomplete image information by identifying contiguous faces of the 3D scene with a side that is less than a predefined amount of another side or that is below a threshold dimension;
reconstructing the detected regions of the 3D scene;
determining replacement information and modify the reconstructed regions; and
rendering the 3D scene with the modified reconstructed regions.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
identifying holes in the rendered 3D scene from one or more viewpoints; and
refining the rendered 3D scene to fill in the holes.

### Claim 11 (depends on 9)

The method of claim 9, wherein the first 2D image is obtained from a first imager and the second 2D image is obtained from a second imager.

### Claim 12 (depends on 9)

The method of claim 9, wherein the determining comprises:
identifying a boundary surrounding each detected region;
identifying background information in the boundary surrounding each detected region;
identifying foreground information in the boundary surrounding each detected region; and
blending the background and foreground information through each detected region giving the background information higher weight than the foreground information.

### Claim 13 (depends on 12)

The method of claim 12, wherein the blending comprises:
diffusing missing information from the background information through each detected region into the foreground information.

### Claim 14 (depends on 9)

The method of claim 9, wherein the creating the 3D scene comprises creating a depth map including pixel vertices and corresponding image information from the first and second 2D images and connecting the vertices to form first faces, and wherein the reconstructing further includes:
connecting the vertices of boundary regions to form second faces, wherein the second faces are different than the first faces.

### Claim 15 (depends on 9)

The method of claim 9, wherein the predefined amount is five percent.

### Claim 16 (depends on 9)

The method of claim 9, wherein the the threshold dimension is one millimeter.

### Claim 17 (depends on 9)

The method of claim 9, wherein the creating the 3D scene comprises creating a depth map including pixel vertices and corresponding image information from the first and second 2D images and connecting the vertices to form multi-angular faces, and wherein the detecting comprises:
identifying contiguous faces including low confidence faces having at least one vertex generated with inconsistent values exceeding a threshold value between the first 2D image and the second 2D image.

### Claim 18

A non-transitory computer readable medium including instructions for creating a three-dimensional (3D) scene when executed by a processor, the instructions comprising:
obtaining a first two-dimensional (2D) image of a scene object from a first viewpoint;
obtaining a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint;
creating a 3D scene from the first and second 2D images;
detecting regions of the 3D scene with incomplete image information by identifying contiguous faces of the 3D scene with a side that is less than a predefined amount of another side or that is below a threshold dimension;
reconstructing the detected regions of the 3D scene;
determining replacement information and modify the reconstructed regions; and
rendering the 3D scene with the modified reconstructed regions.

### Claim 19 (depends on 18)

The medium of claim 18, wherein the instructions further comprise:
identifying holes in the rendered 3D scene from one or more viewpoints; and
refining the rendered 3D scene to fill in the holes.

### Claim 20 (depends on 18)

The medium of claim 18, wherein the determining comprises:
identifying a boundary surrounding each detected region;
identifying background information in the boundary surrounding each detected region;
identifying foreground information in the boundary surrounding each detected region; and
blending the background and foreground information through each detected region giving the background information higher weight than the foreground information.
