# US 2021/0082178 A1 · Method and apparatus for processing a 3d scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082178 A1
- Document ID: US20210082178A1
- Status: Published application
- Kind: A1
- Application number: 16/980,319
- Filed: 2019-03-08
- Published: 2021-03-18
- Earliest claimed: 2018-03-13
- Assignees: InterDigital CE Patent Holdings, SAS
- Applicants: Interdigital Ce Patent Holdings
- Inventors: Philippe ROBERT; Salma JIDDI; Tao LUO
- Web page: https://patentgrep.com/patent/US20210082178A1

## CPC

- G06T 15/506: {Illumination models}
- G06T 15/60: Shadow generation
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/74: {involving reference images or patches}
- G06T 15/20: Perspective computation
- G06T 2207/30241: Trajectory
- G06T 2215/12: Shadow map, environment map
- G06T 2215/16: Using real world measurements to influence rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method and an apparatus for processing a 3D scene are disclosed. At least one virtual reference viewpoint in the 3D scene is determined (41). A map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera positioned at a viewpoint distinct from the virtual reference viewpoint is obtained (42), said map of real cast shadows being registered with regards to the virtual reference viewpoint. Parameters for at least one light source in the 3D scene are determined (44) using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint.

## Claims

### Claim 1

A method for processing a 3D scene, comprising:
determining at least one virtual reference viewpoint in the 3D scene,
obtaining a map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera positioned at a viewpoint distinct from the virtual reference viewpoint, said map of real cast shadows being registered with regards to the virtual reference viewpoint, and
determining parameters for at least one light source in the 3D scene using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint.

### Claim 2

An apparatus for processing a 3D scene comprising:
means for determining at least one virtual reference viewpoint in the 3D scene,
means for obtaining a map of registered real cast shadows of objects in the 3D scene from an input image captured by the camera positioned at a viewpoint distinct from the virtual reference viewpoint, said map of real cast shadows being registered with regards to the virtual reference viewpoint, and
means for determining parameters for at least one light source in the 3D scene using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint.

### Claim 3 (depends on 1)

The method according to claim 1, wherein obtaining the map of registered real cast shadows comprises:
detecting the real cast shadows in the input image to obtain a map of detected real cast shadows, and
registering the map of detected real cast shadows with regards to the virtual reference viewpoint.

### Claim 4 (depends on 1)

The method according to claim 1, wherein obtaining the map of registered real cast shadows comprises:
registering the input image with regards to the virtual reference viewpoint to obtain a registered image, and
detecting the real cast shadows in the registered image.

### Claim 5 (depends on 1)

The method according to claim 1, wherein using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint, comprises matching the map of registered real cast shadows and the at least one map of virtual shadows and selecting the map of virtual shadows that best matches the map of registered real cast shadows.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the virtual reference viewpoint corresponds to a viewpoint from which a real scene represented by the 3D scene has been previously captured by the camera.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the virtual reference viewpoint in the 3D scene is determined as being a viewpoint for which a number of pixels overlapping in an image captured from the virtual reference viewpoint and an image captured from the viewpoint of the camera is above a determined value.

### Claim 8 (depends on 1)

The method according to claim 1, wherein when the virtual reference viewpoint and the camera's viewpoint do not satisfy a determined criterion, a new virtual reference viewpoint is determined.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the virtual reference viewpoint and the camera's viewpoint do not satisfy the determined criterion if a number of pixels overlapping in an image captured from the virtual reference viewpoint and an image captured from the camera's viewpoint is below a determined value.

### Claim 10 (depends on 8)

The method according to claim 8, further comprising:
comparing the poses of the virtual reference viewpoint and of the new virtual reference viewpoint with the camera poses, and
selecting among the virtual reference viewpoint and the new virtual reference viewpoint, the virtual reference viewpoint closest to the camera's viewpoint,
light source parameters in the 3D scene being determined with regards to the selected virtual reference viewpoint.

### Claim 11 (depends on 1)

The method according to claim 1, wherein determining said at least one virtual reference viewpoint in the 3D scene, comprises:
detecting movement of the camera capturing the real scene, and
predicting a camera trajectory in the 3D scene as a function of the detected movement,
said at least one virtual reference viewpoint being determined according to the predicted camera trajectory.

### Claim 12 (depends on 1)

The method according to claim 1, wherein determining said at least one virtual reference viewpoint in the 3D scene comprises determining and storing at least one shadow map associated to a light source of a 3D scene.

### Claim 13 (depends on 1)

The method according to claim 1, further comprising:
detecting geometry changes in the 3D scene, and
determining areas in the input image captured by the camera that are impacted by the geometry changes,
said determined areas being discarded when determining parameters for at least one light source in the 3D scene using the virtual reference viewpoint until virtual shadow maps associated to the virtual reference viewpoint are updated based on the geometry changes.

### Claim 14

(canceled)

### Claim 15 (depends on 2)

The apparatus according to claim 2, wherein obtaining the map of registered real cast shadows comprises:
detecting the real cast shadows in the input image to obtain a map of detected real cast shadows, and
registering the map of detected real cast shadows with regards to the virtual reference viewpoint.

### Claim 16 (depends on 2)

The apparatus according to claim 2, wherein obtaining the map of registered real cast shadows comprises:
registering the input image with regards to the virtual reference viewpoint to obtain a registered image, and
detecting the real cast shadows in the registered image.

### Claim 17 (depends on 2)

The apparatus according to claim 2, wherein using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint, comprises matching the map of registered real cast shadows and the at least one map of virtual shadows and selecting the map of virtual shadows that best matches the map of registered real cast shadows.

### Claim 18 (depends on 2)

The apparatus according to claim 2, wherein the virtual reference viewpoint corresponds to a viewpoint from which a real scene represented by the 3D scene has been previously captured by the camera.

### Claim 19 (depends on 2)

The apparatus according to claim 2, wherein the virtual reference viewpoint in the 3D scene is determined as being a viewpoint for which a number of pixels overlapping in an image captured from the virtual reference viewpoint and an image captured from the viewpoint of the camera is above a determined value.

### Claim 20

A non-transitory computer-readable medium storing instructions operative, when executed by a processor, to perform at least:
determining at least one virtual reference viewpoint in the 3D scene,
obtaining a map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera positioned at a viewpoint distinct from the virtual reference viewpoint, said map of real cast shadows being registered with regards to the virtual reference viewpoint, and
determining parameters for at least one light source in the 3D scene using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint.
