# US 2015/0146972 A1 · Predicting a light probe for an outdoor image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0146972 A1
- Document ID: US20150146972A1
- Status: Published application
- Kind: A1
- Application number: 14/091,270
- Filed: 2013-11-26
- Published: 2015-05-28
- Earliest claimed: 2013-11-26
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: JEAN-FRANCOIS LALONDE; IAIN MATTHEWS
- Web page: https://patentgrep.com/patent/US20150146972A1

## CPC

- G06T 15/506: {Illumination models}
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 20/38: {Outdoor scenes}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for predicting light probes for outdoor images are disclosed. A light probe database is created to learn a mapping from the outdoor image's features to predicted outdoor light probe illumination parameters. The database includes a plurality of images, image features for each of the plurality of images, and a captured light probe for each of the plurality of images. A light probe illumination model based on a sun model and sky model is fitted to the captured light probes. The light probe for the outdoor image may be predicted based on the database dataset and fitted light probe models.

## Claims

### Claim 1

A computer-implemented method, comprising predicting a light probe for an outdoor image based on a learned mapping from the outdoor image's features to light probe illumination parameters.

### Claim 2 (depends on 1)

The method of claim 1, wherein the learned mapping is based on a light probe database comprising:
a plurality of images;
image features for each of the plurality of images; and
a plurality of captured light probes associated with the plurality of images.

### Claim 3 (depends on 2)

The method of claim 2, wherein the outdoor image's features comprise features computed from a surface normal.

### Claim 4 (depends on 3)

The method of claim 3, wherein image features for each of the plurality of images in the light probe database comprise a normal map.

### Claim 5 (depends on 2)

The method of claim 2, wherein the plurality of images comprise a plurality of locations captured under a plurality of illumination conditions.

### Claim 6 (depends on 5)

The method of claim 5, wherein the light probe illumination parameters comprise sun parameters and sky parameters.

### Claim 7 (depends on 6)

The method of claim 6, wherein the learned mapping is based on a sun model for the sun parameters and a sky model for the sky parameters.

### Claim 8 (depends on 7)

The method of claim 7, wherein the sun parameters comprise sun position, sun angular variation, and sun color.

### Claim 9 (depends on 8)

The method of claim 8, wherein the sky parameters comprise sky color and sky angular variation.

### Claim 10 (depends on 7)

The method of claim 7, wherein the sun model is based on a double exponential model.

### Claim 11 (depends on 7)

The method of claim 7, wherein the sky model is based on one of a Preetham model, a cosine model, or a von-Mises Fisher model.

### Claim 12 (depends on 7)

The method of claim 7, wherein the sun model and the sky model are fitted to the light probe database captured light probes to create fitted models.

### Claim 13 (depends on 7)

The method of claim 7, further comprising inserting a virtual object into the outdoor image and lighting the virtual object using the predicted light probe.

### Claim 14 (depends on 9)

The method of claim 9, wherein predicting a light probe for an outdoor image comprises:
predicting the sun position based on a probabilistic model; and
predicting the sun angular variation, sun colors, sky colors, and sky angular variation based on a regression technique.

### Claim 15 (depends on 14)

The method of claim 14, wherein the sun position is predicted relative to the camera that took the outdoor image.

### Claim 16 (depends on 14)

The method of claim 14, wherein the regression technique is a kernel regression or a linear regression.

### Claim 17

A system, comprising:
a camera configured to take an outdoor image; and
a computer configured to predict a light probe for the outdoor image based on a learned mapping from the outdoor image's features to light probe illumination parameters.

### Claim 18 (depends on 17)

The system of claim 17, wherein the learned mapping is based on a light probe database comprising:
a plurality of images;
image features for each of the plurality of images; and
a plurality of captured light probes associated with the plurality of images.

### Claim 19 (depends on 16)

The system of claim 16, wherein the outdoor image's features comprises features computed from a surface normal.

### Claim 20 (depends on 19)

The system of claim 19, wherein image features for each of the plurality of images in the light probe database comprise a normal map.

### Claim 21 (depends on 18)

The system of claim 18, wherein the plurality of images comprise a plurality of locations captured under a plurality of illumination conditions.

### Claim 22 (depends on 21)

The method of claim 21, wherein the light probe illumination parameters comprise sun parameters and sky parameters.

### Claim 23 (depends on 22)

The system of claim 22, wherein the learned mapping is based on a sun model for the sun parameters and a sky model for the sky parameters.

### Claim 24 (depends on 23)

The system of claim 23, wherein the sun parameters comprise sun position, sun angular variation, and sun color.

### Claim 25 (depends on 24)

The system of claim 24, wherein the sky parameters comprise sky color and sky angular variation.

### Claim 26 (depends on 23)

The system of claim 23, wherein the sun model is based on a double exponential model.

### Claim 27 (depends on 23)

The system of claim 23, wherein the sky model is based on one of a Preetham model, a cosine model, or a von-Mises Fisher model.

### Claim 28 (depends on 23)

The system of claim 23, wherein the sun model and the sky model are fitted to the light probe database captured light probes to create fitted models.

### Claim 29 (depends on 23)

The system of claim 23, wherein the computer is further configured to:
insert a virtual object into the outdoor image; and
light the virtual object using the predicted light probe.

### Claim 30 (depends on 25)

The system of claim 25, wherein predicting a light probe for an outdoor image comprises:
predicting the sun position based on a probabilistic model; and
predicting the sun angular variation, sun colors, sky colors, and sky angular variation based on a regression technique.

### Claim 31 (depends on 30)

The system of claim 30, wherein the sun position is predicted relative to the camera that took the outdoor image.

### Claim 32 (depends on 30)

The system of claim 30, wherein the regression technique is a kernel regression or a linear regression.

### Claim 33

A method of creating a light probe database, comprising:
capturing a plurality of images for each of a plurality of locations with a camera;
capturing the light probes for each of the plurality of images; and
recovering image features for each of the plurality of captured images.

### Claim 34 (depends on 32)

The method of claim 32, wherein recovering image features for each of the plurality of captured images comprises recovering a normal map for each of the plurality of captured images.

### Claim 35 (depends on 33)

The method of claim 33, wherein the plurality of images comprise a plurality of locations captured under a plurality of illumination conditions.

### Claim 36 (depends on 34)

The method of claim 34, further comprising calibrating and aligning the captured images and captured light probes.

### Claim 37 (depends on 35)

The method of claim 35, wherein the light probe database is used to predict a light probe for an outdoor image.