# US 2023/0410337 A1 · Method and system for scene image modification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410337 A1
- Document ID: US20230410337A1
- Status: Published application
- Kind: A1
- Application number: 18/337,637
- Filed: 2023-06-20
- Published: 2023-12-21
- Earliest claimed: 2019-11-12
- Assignees: Geomagical Labs, Inc.
- Applicants: Geomagical Labs, Inc.
- Inventors: Brian Pugh; Angus Dorbie; Salma Jiddi; Qiqin Dai; Paul Gauthier; Marc Eder; Jianfeng Yin; Luis Puig Morales; Michael Otrada; Konstantinos Nektarios Lianos; Philip Guindi; Brian Totty
- Web page: https://patentgrep.com/patent/US20230410337A1

## CPC

- G06T 7/11: Region-based segmentation
- G06T 7/543: from line drawings
- G06T 7/13: Edge detection
- G06T 7/194: involving foreground-background segmentation
- G06T 7/55: from multiple images
- G06T 7/60: Analysis of geometric attributes
- G06T 7/73: using feature-based methods
- G06T 7/90: Determination of colour characteristics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 2200/04: involving 3D image data
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

System and method for rendering virtual objects onto an image.

## Claims

### Claim 1

A method comprising: with an image processing platform:
accessing a photorealistic image of a room;
generating at least one segmentation mask that identifies real objects included in the photorealistic image;
generating geometric surface information for the photorealistic image;
generating a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and
providing at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, with the end user application:
rendering at least one virtual object onto the photorealistic image by using the dense depthmap, at least one segmentation mask, and the geometric surface information.

### Claim 3 (depends on 2)

The method of claim 2, wherein rendering at least one virtual object onto the photorealistic image comprises: for each virtual object pixel of the virtual object:
determining a depth of the virtual object pixel;
determining a depth of an image pixel at the same location as the virtual object pixel;
in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and
in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

### Claim 4 (depends on 3)

The method of claim 3,
wherein the photorealistic image is bound to a rectangular occlusion quad of the user device,
wherein the occlusion quad is associated with the dense depthmap.
wherein a fragment shader included in the user device processes image candidate pixels from the photorealistic image on the occlusion quad and virtual object candidate pixels from the virtual objects, and
wherein, for each virtual object pixel, determining the depth of the image pixel at the same location as the virtual object pixel comprises: for each virtual object candidate pixel, accessing the depth of the image candidate pixel from the occlusion quad at the same location as the virtual object pixel, by using the dense depthmap associated with the occlusion quad.

### Claim 5 (depends on 2)

The method of claim 2, wherein rendering at least one virtual object onto the photorealistic image comprises: for each virtual object pixel of the virtual object:
determining a depth of the virtual object pixel;
sampling the dense depthmap to generate depthmap samples;
generating a triangular occlusion mesh;
tesselating the triangular occlusion mesh, such that the depths of the pixels of the triangular mesh correspond to depths of the depthmap samples;
registering the triangular occlusion mesh with a 3D graphics system of the user device as an invisible, z-occluding scene model; and
using the graphics system to render the virtual objects with occlusion by the triangular occlusion mesh,
wherein rendering the virtual objects with occlusion by the triangular occlusion mesh comprises: discarding occluded virtual object pixels, and compositing non-occluded virtual object pixels with the photorealistic image.

### Claim 6 (depends on 4)

The method of claim 4, further comprising: updating the dense depthmap after rendering the virtual objects with occlusion, and providing an updated dense depthmap to the end user application.

### Claim 7 (depends on 4)

The method of claim 4,
further comprising:
generating an updated dense depthmap; and
rendering at least one virtual object onto the photorealistic image by using the at least one segmentation mask, the updated dense depthmap, and the geometric surface information,
wherein rendering at least one virtual object onto the photorealistic image comprises: for each virtual object pixel of the virtual object:
determining a depth of the virtual object pixel;
determining a depth of an image pixel at the same location as the virtual object pixel;
in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and
in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the dense depthmap comprises:
identifying edge pixels included in edges of real objects included in the photorealistic image, by using at least one generated segmentation mask;
for at least one edge pixel, enhancing the depth for the pixel based on depths of other edge pixels included in the same edge;
identifying planar surface pixels that are included in a planar surface identified by the generated geometric surface information; and
for at least one planar surface pixel, adjusting the depth for the pixel based on depths of other planar surface pixels included in the same planar surface.

### Claim 9 (depends on 2)

The method of claim 2, wherein rendering at least one virtual object onto the photorealistic image comprises: for each virtual object pixel of the virtual object:
identifying placeability behavior of the virtual object;
mapping two-dimensional user interface pointer coordinates to three-dimensional scene coordinates based on the placeability behavior of the virtual object and nearby scene geometry;
determining a depth of the virtual object pixel;
determining a depth of an image pixel at the same location as the virtual object pixel;
in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and
in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, performing placement processing for the virtual object pixel.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving user selection information identifying selected portions of at least one real object to be removed from the photorealistic image; and
for each pixel in the selected portions:
estimating a new depth for the pixel and updating the dense depthmap to include the new depth, and
determining a replacement color for the pixel and updating the photorealistic image to include the determined color.

### Claim 11 (depends on 10)

The method of claim 10,
further comprising: for each real object to be removed, reducing at least one cast shadow associated with the real object to be removed,
wherein estimating a new depth for a pixel of a selected portion comprises: updating the dense depthmap to include a predetermined depth as the new depth for the pixel, and
wherein reducing at least one cast shadow comprises at least one of:
inferring at least one cast shadow by performing image processing,
inferring at least one cast shadow by using a trained neural network,
inferring at least one cast shadow from detected and estimated light sources,
inferring at least one cast shadow from intrinsic image decomposition, and
inferring at least one cast shadow from plenoptic light fields.

### Claim 12 (depends on 10)

The method of claim 10, wherein estimating a new depth for a pixel of a selected portion comprises:
identifying a room architectural geometric surface that includes the location of the pixel, by using the generated geometric surface information for the photorealistic image, wherein the generated geometric surface information identifies at least one room architectural geometric surface included in the photorealistic image;
identifying at least one depth of the room architectural geometric surface, by using the generated dense depthmap;
computing a depth for the pixel by using the identified at least one depth of the room architectural geometric surface; and
updating the dense depthmap to include the computed depth as the new depth for the pixel.

### Claim 13 (depends on 10)

The method of claim 10, wherein estimating a new depth for a pixel of a selected portion comprises:
estimating a depth of an obscured geometric surface obscured by the selected portion, by using the dense depthmap to identify depths of the real objects surrounding the selected portion; and
updating the dense depthmap to include the estimated depth of the obscured geometric surface as the new depth for the pixel.

### Claim 14 (depends on 10)

The method of claim 10, wherein estimating a new depth for a pixel of a selected portion comprises:
estimating a depth of an obscured geometric surface obscured by the selected portion, by using trained neural networks and geometric priors from the dense depthmap; and
updating the dense depthmap to include the estimated depth of the obscured geometric surface as the new depth for the pixel.

### Claim 15 (depends on 10)

The method of claim 10,
wherein determining a replacement color for a pixel of a selected portion comprises: assigning to the pixel one of: a static ghosting color, a dynamic ghosting color, a static ghosting pattern, and a dynamic ghosting pattern.

### Claim 16 (depends on 10)

The method of claim 10,
wherein determining a replacement color for a pixel of a selected portion comprises:
identifying user selection of a color, and assigning the selected color to the pixel,
wherein the user selection of the color relates to one of: the pixel, and a patch of pixels that includes the pixel that is assigned the color.

### Claim 17 (depends on 10)

The method of claim 10,
wherein determining a replacement color for a pixel of a selected portion comprises: assigning a color of a transparency grid pattern to the pixel.

### Claim 18 (depends on 17)

The method of claim 17,
wherein the transparency grid pattern is perspective warped based on inferred geometry behind the selected portion.

### Claim 19 (depends on 10)

The method of claim 10,
wherein determining a replacement color for a pixel of a selected portion comprises:
performing texture synthesis to identify a texture of pixels likely surrounding the pixel;
identifying a replacement color for the pixel by using the identified texture, and assigning the replacement color to the pixel.

### Claim 20 (depends on 10)

The method of claim 10,
wherein determining a replacement color for a pixel of a selected portion comprises at least one of:
generating a replacement virtual object that includes the location of the pixel, wherein the replacement virtual object includes pixels having various colors that represent a synthetic texture; and
estimating a plenoptic light field for a scene of the photorealistic image, and using the light field to re-generate an image area affected by removal of pixels included in the selected portion.
