# US 2012/0268571 A1 · Multiview face capture using polarized spherical gradient illumination

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0268571 A1
- Document ID: US20120268571A1
- Status: Published application
- Kind: A1
- Application number: 13/449,729
- Filed: 2012-04-18
- Published: 2012-10-25
- Earliest claimed: 2011-04-19
- Assignees: University of Southern California
- Inventors: PAUL E. DEBEVEC; ABHIJEET GHOSH; GRAHAM FYFFE
- Web page: https://patentgrep.com/patent/US20120268571A1

## CPC

- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- G06T 7/586: from multiple light sources, e.g. photometric stereo
- G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings
- G06V 40/166: {using acquisition arrangements}
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10152: Varying illumination
- G06T 2207/30201: Face
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A multiview face capture system may acquire detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints. A lighting system may illuminate a face with polarized light from multiple directions. The light may be polarized substantially parallel to a reference axis during a parallel polarization mode of operation and substantially perpendicular to the reference axis during a perpendicular polarization mode of operation. Multiple cameras may each capture an image of the face along a materially different optical axis and have a linear polarizer configured to polarize light traveling along its optical axis in a direction that is substantially parallel to the reference axis. A controller may cause each of the cameras to capture an image of the face while the lighting system is in the parallel polarization mode of operation and again while the lighting system is in the perpendicular polarization mode of operation.

## Claims

### Claim 1

A multiview face capture system for acquiring detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints, the system comprising:
a lighting system configured to illuminate a face from multiple directions with light that is polarized:
substantially parallel to a reference axis during a parallel polarization mode of operation; and
substantially perpendicular to the reference axis during a perpendicular polarization mode of operation;
a camera system that includes one or more cameras, each configured to capture an image of the face along a materially different optical axis and each having a linear polarizer configured to polarize light traveling along its optical axis in a direction that is substantially parallel to the reference axis; and
a controller configured to control the lighting system and/or the camera system so as to cause each of the one or more cameras to capture an image of the face while the lighting system is in the parallel polarization mode of operation and again while the lighting system is in the perpendicular polarization mode of operation.

### Claim 2 (depends on 1)

The multiview face capture system of claim 1 wherein:
the lighting system is configured to cause, during a spherical mode of operation, the illumination of the face during both polarization modes of operation to be substantially spherical; and
the controller is configured to control the lighting system and/or the camera system so as to cause each of the one or more cameras to capture an image of the face during the spherical mode of operation while the lighting system is in the parallel polarization mode of operation and again while the lighting system is in the perpendicular polarization mode of operation.

### Claim 3 (depends on 1)

The multiview face capture system of claim 1 wherein:
the lighting system is configured to cause, during a first gradient mode of operation, the intensity of the illumination of the face to vary in a gradient across a first directional axis during both polarization modes of operation; and
the controller is configured to control the lighting system and/or the camera system so as to cause each of the one or more cameras to capture an image of the face during the first gradient mode of operation while the lighting system is in the parallel polarization mode of operation and again while the lighting system is in the perpendicular polarization mode of operation.

### Claim 4 (depends on 3)

The multiview face capture system of claim 3 wherein:
the lighting system is configured to cause, during a second gradient mode of operation, the intensity of the illumination of the face to vary in a gradient across a second gradient axis that is substantially perpendicular to the first gradient axis during both polarization modes of operation; and
the controller is configured to control the lighting system and/or the camera system so as to cause each of the one or more cameras to capture an image of the face during each of the first and the second gradient modes of operation while the lighting system is in the parallel polarization mode of operation and again while the lighting system is in the perpendicular polarization mode of operation.

### Claim 5 (depends on 3)

The multiview face capture system of claim 3 wherein:
the lighting system is configured to cause, during a third gradient mode of operation, the intensity of the illumination of the face to vary in a gradient across a third gradient axis that is substantially perpendicular to the first and the second gradient axes during both polarization modes of operation; and
the controller is configured to control the lighting system and/or the camera system so as to cause each of the one or more cameras to capture an image of the face during each of the first, second, and third gradient modes of operation while the lighting system is in the parallel polarization mode of operation and again while the lighting system is in the perpendicular polarization mode of operation.

### Claim 6 (depends on 1)

The multiview face capture system of claim 1 wherein the lighting system includes a light source that generates the illumination from each of the multiple directions that is physically separate from the light source that generates the illumination from each of the other multiple directions.

### Claim 7 (depends on 1)

The multiview face capture system of claim 1 wherein the lighting system includes a flat panel of separately-controllable pixels.

### Claim 8 (depends on 1)

The multiview face capture system of claim 1 wherein each optical axis is substantially perpendicular to the reference axis.

### Claim 9 (depends on 1)

The multiview face capture system of claim 1 wherein each optical axis passes through the approximate vertical center of the face.

### Claim 10 (depends on 1)

The multiview face capture system of claim 1 wherein the camera system includes multiple cameras, each configured to capture an image of the face along a materially different optical axis and each having a linear polarizer configured to polarize light traveling along its optical axis in a direction that is substantially parallel to the reference axis.

### Claim 11 (depends on 10)

The multiview face capture system of claim 10 wherein each optical axis is substantially perpendicular to the reference axis.

### Claim 12 (depends on 1)

The multiview face capture system of claim 1 wherein each optical axis passes through the approximate vertical center of the face.

### Claim 13

Non-transitory, tangible, computer-readable storage media containing a program of instructions configured to cause a computer system running the program of instructions to generate data representative of 3D geometry of a face based at least in part on data that is representative of a specular surface normal map of the face from multiple viewpoints.

### Claim 14 (depends on 13)

The media of claim 13 wherein the program of instructions causes the computer system running the program of instructions to generate the data representative of the 3D geometry of the face also based on a specular albedo map of the face from multiple viewpoints.

### Claim 15 (depends on 13)

The media of claim 13 wherein the program of instructions causes the computer system running the program of instructions to generate the data representative of the 3D geometry of the face also based on a diffuse albedo map of the face from multiple viewpoints.

### Claim 16 (depends on 13)

The media of claim 13 wherein the program of instructions causes the computer system running the program of instructions to generate the data representative of the 3D geometry of the face also based on a specular albedo map of the face and a diffuse albedo map of the face, each from multiple viewpoints.

### Claim 17 (depends on 13)

The media of claim 13 wherein the program of instructions causes the computer system running the program of instructions to compute a cylindrical displacement map X that minimizes substantially the following graphical cost function:
E

(
X
)
=
∑
s
∈
V


φ
s

(
x
s
)
+
∑
(
s
,
t
)
∈
E


ψ
st

(
x
s
,
x
t
)
where V is a set of all pixel sites in the displacement map X, E is a set of edges connecting neighboring sites, xs is a displacement distance from cylinder axis at site s, φs, is a data term, and φst is a smoothing term.

### Claim 18

A face capture system for acquiring detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints, the system comprising:
a lighting system configured to illuminate a face from multiple directions with polarized light that is polarized:
substantially parallel to a reference axis during a parallel polarization mode of operation; or
substantially perpendicular to the reference axis during a perpendicular polarization mode of operation; and
a camera system that includes one or more cameras, each configured to capture an image of the face along a materially different optical axis and each having a linear polarizer configured to polarize light traveling along its optical axis in a direction that is substantially parallel to the reference axis while the lighting system is in the parallel polarization mode of operation or the perpendicular polarization mode of operation.

### Claim 19 (depends on 18)

The face capture system of claim 18 wherein the camera system includes multiple cameras, each configured to capture an image of the face along a materially different optical axis and each having a linear polarizer configured to polarize light traveling along its optical axis in a direction that is substantially parallel to the reference axis.

### Claim 20 (depends on 18)

The multiview face capture system of claim 18 wherein the lighting system is configured to cause, during a spherical mode of operation, the illumination of the face to be substantially spherical.
