# US 2008/0187213 A1 · Fast Landmark Detection Using Regression Methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0187213 A1
- Document ID: US20080187213A1
- Status: Published application
- Kind: A1
- Application number: 11/671,760
- Filed: 2007-02-06
- Published: 2008-08-07
- Earliest claimed: 2007-02-06
- Assignees: Microsoft Corporation
- Inventors: Cha Zhang; Paul Viola; Sang Min Oh
- Web page: https://patentgrep.com/patent/US20080187213A1

## CPC

- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A landmark detection technique that can quickly detect both objects of interest and landmarks within the objects in an image using regression methods. The present fast landmark detection scheme reuses existing feature values used for object detection (e.g., face detection) to find the landmarks in an object (e.g., the eyes and mouth of the face). Hence, the technique provides landmark detection functionality at almost no cost.

## Claims

### Claim 1

A computer-implemented process for detecting landmarks and their positions in an object detected in an input image, comprising using a computer to perform the following process actions:
creating a database comprising a plurality of training feature characterizations, each of which characterizes features of an object in an image;
for each object in the database computing landmark features that define the object and defining the ground truth locations of these landmark features;
training a regressor using a regression learning procedure to learn a relationship that defines the location of landmarks in any detected object given said feature characterizations;
inputting a portion of an input image into an object detector and outputting the location of any object found in the portion of the input image and feature characterizations used to find any object found;
inputting the feature characterizations and the location of any object found in the portion of the input image to into the trained regressor to output the landmark locations.

### Claim 2 (depends on 1)

The computer-implemented process of claim 1 wherein the regression learning procedure comprises employing linear regression.

### Claim 3 (depends on 1)

The computer-implemented process of claim 1 wherein the regression learning procedure comprises employing a neural network.

### Claim 4 (depends on 1)

The computer-implemented process of claim 1 wherein the regression learning procedure comprises employing additive polynomial modeling.

### Claim 5 (depends on 1)

The computer-implemented process of claim 1 wherein the regression learning procedure comprises employing a regression tree.

### Claim 6 (depends on 1)

The computer-implemented process of claim 1 wherein the feature characterizations are raw feature values output from the object detector.

### Claim 7 (depends on 1)

The computer-implemented process of claim 1 wherein the feature characterizations are transformed feature values output from the object detector.

### Claim 8 (depends on 1)

The computer-implemented process of claim 1 wherein the feature characterizations are raw pixel values output from the object detector.

### Claim 9 (depends on 1)

The computer-implemented process of claim 1 wherein the object detector is a face detector and wherein the landmarks are the eyes, nose and mouth of any face detected by the face detector.

### Claim 10 (depends on 1)

A computer-readable medium having computer-executable instructions for performing the process recited in claim 1.

### Claim 11

A system for locating landmarks in an object detected by an object detector, comprising:
a general purpose computing device;
a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,
input an object in an image detected by an object detector that employs features to detect the object, and the features used to detect the object, into a regressor trained to find the locations of landmarks in the object; and
output the locations of the landmarks in the object.

### Claim 12 (depends on 11)

The system of claim 11 wherein the object detector is a face detector.

### Claim 13 (depends on 11)

The system of claim 11 wherein the regressor is trained using a regression procedure.

### Claim 14 (depends on 13)

The system of claim 13 wherein the regression procedure comprises at least one of:
mean prediction;
linear regression;
a neural network;
additive polynomial modeling;
a regression tree; and
a boosted regression tree.

### Claim 15 (depends on 11)

The system of claim 11 wherein the output landmarks are used for one of:
face pose estimation;
virtual makeup application; and
teleconferencing.

### Claim 16

A computer-implemented process for training a regressor to detect landmarks and their positions in a face detected in an input image and using the trained regressor, comprising using a computer to perform the following process actions:
creating a training database of faces;
for each face in the training database computing landmarks that define the face and marking the ground truth locations of these landmarks; and
training a regressor using a regression learning procedure and the training database with the defined ground truth locations and features used by the face detector to learn a matrix that defines the landmarks in any detected face.

### Claim 17 (depends on 16)

The computer-implemented process of claim 16 further comprising using the trained regressor to define the location of landmarks in a face detected by the face detector, comprising:
inputting a portion of an input image into an face detector and outputting the location of any face found in the portion of the input image and features used to find any face found;
inputting the features and the location of any face found in the portion of the input image to into the trained regressor to output the landmark locations.

### Claim 18 (depends on 17)

The computer-implemented process of claim 17 wherein the regression procedure comprises employing a neural network and wherein the features are pixel values.

### Claim 19 (depends on 17)

The computer-implemented process of claim 17 wherein the regression procedure comprises employing a regression tree and wherein the features are raw or transformed features.

### Claim 20 (depends on 19)

The computer-implemented process of claim 19 wherein the regression tree is a boosted regression tree.