# US 2017/0256208 A1 · Media item relighting technique

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0256208 A1
- Document ID: US20170256208A1
- Status: Published application
- Kind: A1
- Application number: 15/063,373
- Filed: 2016-03-07
- Published: 2017-09-07
- Earliest claimed: 2016-03-07
- Assignees: Ecole Polytechnique Federale De Lausanne (Epfl)
- Applicants: Ecole Polytechnique Federale De Lausanne (Epfl)
- Inventors: Niranjan Thanikachalam; Loic Arnaud Baboulaz; Damien Firmenich; Sabine Suesstrunk; Martin Vetterli
- Web page: https://patentgrep.com/patent/US20170256208A1

## CPC

- G06T 15/50: Lighting effects
- G09G 3/3406: {Control of illumination source (illumination devices structurally associated with liquid crystal cells G02F1/1336)}
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G09G 5/10: Intensity circuits
- G06T 2200/21: involving computational photography
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10152: Varying illumination
- G06T 2207/20048: Transform domain processing
- G09G 2360/144: the light being ambient light
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

The present invention concerns a method of method of relighting a media item comprising media elements. The method comprises, for at least some of the media elements: determining, in a first signal domain, a light transport function describing the appearance of a particular media element under different illumination conditions at least partly defined by positions of a light source used to illuminate the particular media element; sampling, in the first signal domain, the light transport function of the particular media element to obtain discrete data samples of the light transport function; projecting the data samples in the first signal domain into a sampling grid to obtain spatially sparsely and non-uniformly sampled light transport function; interpolating, in a second signal domain, the sparsely and non-uniformly sampled light transport function to obtain an approximate light transport matrix in the second signal domain; converting the approximate light transport matrix into the first signal domain to obtain an approximate substantially uniformly sampled light transport function in the first signal domain; and using, in the first signal domain, the approximate substantially uniformly light transport function to relight the media item.

## Claims

### Claim 1

A method of relighting a media item comprising media elements, the method comprising, for at least some of the media elements:
determining, in a first signal domain, a light transport function describing the appearance of a particular media element under different illumination conditions at least partly defined by positions of a light source used to illuminate the particular media element;
sampling, in the first signal domain, the light transport function of the particular media element to obtain discrete data samples of the light transport function;
projecting the data samples, in the first signal domain, into a sampling grid to obtain a spatially sparsely and non-uniformly sampled light transport function;
interpolating, in a second signal domain, the sparsely and non-uniformly sampled light transport function to obtain a light transport matrix in the second signal domain; and
transforming the light transport matrix into the first signal domain to obtain a substantially uniformly sampled light transport function for relighting the media item.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the light source used to illuminate the media item moves along a trajectory.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the light source moves in a plane.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the light source moves within an illumination region, and the movement within the illumination region is random.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the first domain is a spatial domain, and the second domain is a frequency domain.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the different illumination conditions are also at least partly defined by a light source intensity value in a particular light source position.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the interpolation comprises performing, for the at least some of the media elements, compressive sensing of the sparsely and non-uniformly sampled light transport function.

### Claim 8 (depends on 1)

The method according claim 1, wherein the media item is an image or a video.

### Claim 9 (depends on 1)

The method according claim 1, wherein the media element is a pixel or a voxel.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the method further comprises:
illuminating at least a portion of the media item by a light source of a first apparatus by moving the first apparatus along a trajectory;
tracking the trajectory;
recording media item frames of the media item with a second apparatus; and
determining a position of the light source for at least some of the media item frames to obtain the light transport function.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the second apparatus is stationary.

### Claim 12 (depends on 10)

The method according to claim 10, wherein the first and second apparatuses are handheld electronic devices.

### Claim 13 (depends on 10)

The method according to claim 10, wherein the tracking is carried out by a parallel tracking and mapping algorithm.

### Claim 14 (depends on 10)

The method according to claim 10, wherein the method further comprises the first apparatus gradually displaying the trajectory while the light source moves along the trajectory.

### Claim 15 (depends on 10)

The method according to claim 10, wherein the method further comprises a user of the first apparatus selecting the media item by the first apparatus.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the method further comprises, in response to the selection, the first apparatus indicates a region comprising the media item to be illuminated by the light source of the first apparatus.

### Claim 17 (depends on 1)

The method according to claim 1, wherein the method further comprises using, in the first signal domain, the substantially uniformly light transport function to relight the media item.

### Claim 18 (depends on 1)

A computer program product comprising instructions for implementing the steps of the method according to claim 1 when loaded and run on computing means of an electronic device.

### Claim 19

A media item processing apparatus for relighting a media item comprising media elements, the computing apparatus comprising means for, for at least some of the media elements:
determining, in a first signal domain, a light transport function describing the appearance of a particular media element under different illumination conditions at least partly defined by positions of a light source used to illuminate the particular media element;
sampling, in the first signal domain, the light transport function of the particular media element to obtain discrete data samples of the light transport function;
projecting the data samples in the first signal domain into a sampling grid to obtain spatially sparsely and non-uniformly sampled light transport function;
interpolating, in a second signal domain, the sparsely and non-uniformly sampled light transport function to obtain a light transport matrix in the second signal domain; and
transforming the light transport matrix into the first signal domain to obtain a substantially uniformly sampled light transport function for media item relighting.
