# US 2017/0286759 A1 · Method and system of facial expression recognition using linear relationships within landmark subsets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286759 A1
- Document ID: US20170286759A1
- Status: Published application
- Kind: A1
- Application number: 15/512,503
- Filed: 2014-10-23
- Published: 2017-10-05
- Earliest claimed: 2014-10-23
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Anbang YAO; Yurong CHEN
- Web page: https://patentgrep.com/patent/US20170286759A1

## CPC

- G06V 40/175: {Static expression}
- G06F 18/2132: based on discrimination criteria, e.g. discriminant analysis
- G06V 10/457: {by analysing connectivity, e.g. edge linking, connected component analysis or slices}
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 40/164: {using holistic features}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/176: {Dynamic expression}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system, article, and method to provide facial expression recognition using linear relationships within landmark subsets.

## Claims

### Claim at position 1

1-25. (canceled)

### Claim 26

A computer-implemented method of facial expression recognition, comprising:
obtaining image data of at least one image that comprises content having at least one person's face;
determining at least one subset of landmarks for at least one main landmark, wherein each landmark has data to indicate information about a facial feature of the face;
determining a relationship among the at least one main landmark and the subset of landmarks; and
using a discriminative model to label a facial expression class for the face by using the relationship.

### Claim 27 (depends on 26)

The method of claim 26 wherein the relationship is a geometrical relationship determined by using linear regression-type calculations.

### Claim 28 (depends on 26)

The method of claim 26 comprising:
forming a plurality of the subsets for a plurality of the main landmarks; and
determining the geometrical relationship for each of the subsets to form a group of the geometrical relationships; and
using the group of geometrical relationships to form a descriptor to use to determine a facial expression classification of the face.

### Claim 29 (depends on 28)

The method of claim 28 wherein each of the main landmarks of the plurality of main landmarks has a single subset as a main landmark while being available to be in other subsets as a neighbor landmark.

### Claim 30 (depends on 28)

The method of claim 28 wherein the subsets overlap so that at least one landmark is in more than one subset.

### Claim 31 (depends on 28)

The method of claim 28 comprising numbering the landmarks on the face to form an index, maintaining the numbering while forming the relationships, and stacking and concatenating the relationships in the same numerical sequence as the index number of the main landmark in order to form the descriptor.

### Claim 32 (depends on 28)

The method of claim 28 wherein each geographical relationship is a vector of geographical relationship values wherein each value corresponds to a landmark in the subset.

### Claim 33 (depends on 32)

The method of claim 32 comprising using the geographical relationship values to form the descriptor by concatenating the geographical relationship values in each subset in a numerical order according to the index numbers of the landmarks corresponding to the geometrical relationship values, and within the space in the descriptor reserved for the subset by the index number of the main landmark.

### Claim 34 (depends on 28)

The method of claim 28 wherein the descriptor is a vector that includes values that are each a difference in geometrical relationship for a main landmark between the values of the landmarks in the subset on a current frame and a corresponding subset on an onset frame.

### Claim 35 (depends on 28)

The method of claim 28 comprising reducing the dimensionality of the descriptor by using primary component analysis (PCA) before using the descriptor in the model.

### Claim 36 (depends on 26)

The method of claim 26 wherein multiple landmarks less than all landmarks of the face each have a subset of neighbor landmarks to use to determine the facial expression of the face.

### Claim 37 (depends on 26)

The method of claim 26 wherein each landmark of the face has a subset of neighbor landmarks used in the calculations.

### Claim 38 (depends on 26)

The method of claim 26 wherein at least one of:
a selected number of closest nearby neighbor landmarks relative to a main landmark are included in the subset of the main landmark, and
any landmark within a certain distance from the main landmark is included in the subset of the main landmark.

### Claim 39 (depends on 26)

The method of claim 26 wherein the membership of landmarks in the subsets remains the same for the same face on multiple frames.

### Claim 40 (depends on 26)

The method of claim 26 comprising:
tracking landmarks on the image data;
normalizing the image data forming the face by using reference face shape data to form an aligned face shape;
extracting features comprising:
using the reference face shape data to determine the subsets, and
using the aligned face shape to determine the relationships; and
using a combination of the relationships in a discriminating label model to provide a face expression classification for the face.

### Claim 41 (depends on 26)

The method of claim 26 wherein a plurality of differences between a current geometrical relationship and a corresponding geometrical relationship at a previous frame is placed into a latent dynamic continuous random field (LDCRF) model to match a facial expression classification to the face.

### Claim 42 (depends on 41)

The method of claim 41 wherein the plurality of differences are combined into a current descriptor, and the LDCRF model determines a facial expression classification for the current face by factoring the current descriptor and at least one descriptor of a previous frame with the same face.

### Claim 43 (depends on 26)

The method of claim 26 wherein the relationship is a geometrical relationship determined by using linear regression-type calculations;
the method comprising:
forming a plurality of the subsets for a plurality of the main landmarks;
determining the geometrical relationship for each of the subsets to form a group of the geometrical relationships; and
using the group of geometrical relationships to form a descriptor to use to determine a facial expression classification of the face;
wherein each of the main landmarks of the plurality of main landmarks has a single subset as a main landmark while being available to be in other subsets as a neighbor landmark;
wherein the subsets overlap so that at least one landmark is in more than one subset;
the method comprising numbering the landmarks on the face to form an index, maintaining the numbering while forming the relationships, and stacking and concatenating the relationships in the same numerical sequence as the index number of the main landmark in order to form the descriptor;
wherein each geographical relationship is a vector of geographical relationship values wherein each value corresponds to a landmark in the subset, the method comprising using the geographical relationship values to form the descriptor by concatenating the geographical relationship values in each subset in a numerical order according to the index numbers of the landmarks corresponding to the geometrical relationship values, and within the space in the descriptor reserved for the subset by the index number of the main landmark;
wherein the descriptor is a vector that includes values that are each a difference in geometrical relationship for a main landmark between the values of the landmarks in the subset on a current frame and a corresponding subset on an onset frame;
the method comprising reducing the dimensionality of the descriptor by using primary component analysis (PCA) before using the descriptor in the model;
wherein at least one of:
multiple landmarks less than all landmarks of the face each have a subset of neighbor landmarks to use to determine the facial expression of the face;
each landmark of the face has a subset of neighbor landmarks used in the calculations;
wherein at least one of:
a selected number of closest nearby neighbor landmarks relative to a main landmark are included in the subset of the main landmark, and
any landmark within a certain distance from the main landmark is included in the subset of the main landmark;
wherein the membership of landmarks in the subsets remains the same for the same face on multiple frames;
the method comprising:
tracking landmarks on the image data;
normalizing the image data forming the face by using reference face shape data to form an aligned face shape;
extracting features comprising:
using the reference face shape data to determine the subsets, and
using the aligned face shape to determine the relationships; and
using a combination of the relationships in a discriminating label model to provide a face expression classification for the face;
wherein a plurality of differences between a current geometrical relationship and a corresponding geometrical relationship at a previous frame is placed into a latent dynamic continuous random field (LDCRF) model to match a facial expression classification to the face; and wherein the plurality of differences are combined into a current descriptor, and the LDCRF model determines a facial expression classification for the current face by factoring the current descriptor and at least one descriptor of a previous frame with the same face.

### Claim 44

A computer-implemented system of facial expression recognition comprising:
a display;
at least one processor communicatively coupled to the display;
at least one memory communicatively coupled to at least one processor and storing image capture data of at least one frame of a video sequence or at least one still photograph; and
at least one facial expression recognition unit communicatively coupled to the processor, and to:
obtain image data of at least one image that comprises content having at least one person's face;
determine at least one subset of landmarks for at least one main landmark, wherein each landmark has data to indicate information about a facial feature of the face;
determine a relationship among the at least one main landmark and the subset of landmarks; and
use a discriminative model to label a facial expression class for the face by using the relationship.

### Claim 45 (depends on 44)

The system of claim 44 wherein the relationship is a geometrical relationship determined by using linear regression-type calculations.

### Claim 46 (depends on 44)

The system of claim 44, the facial expression recognition unit to:
form a plurality of the subsets for a plurality of the main landmarks; and
determine the geometrical relationship for each of the subsets to form a group of the geometrical relationships; and
use the group of geometrical relationships to form a descriptor to use to determine a facial expression classification of the face.

### Claim 47 (depends on 44)

The system of claim 44 wherein the relationship is a geometrical relationship determined by using linear regression-type calculations;
the facial expression recognition unit to:
form a plurality of the subsets for a plurality of the main landmarks;
determine the geometrical relationship for each of the subsets to form a group of the geometrical relationships; and
use the group of geometrical relationships to form a descriptor to use to determine a facial expression classification of the face;
wherein each of the main landmarks of the plurality of main landmarks has a single subset as a main landmark while being available to be in other subsets as a neighbor landmark;
wherein the subsets overlap so that at least one landmark is in more than one subset;
the facial expression recognition unit to number the landmarks on the face to form an index, maintain the numbering while forming the relationships, and stack and concatenate the relationships in the same numerical sequence as the index number of the main landmark in order to form the descriptor;
wherein each geographical relationship is a vector of geographical relationship values wherein each value corresponds to a landmark in the subset, the facial expression recognition unit to use the geographical relationship values to form the descriptor by concatenating the geographical relationship values in each subset in a numerical order according to the index numbers of the landmarks corresponding to the geometrical relationship values, and within the space in the descriptor reserved for the subset by the index number of the main landmark;
wherein the descriptor is a vector that includes values that are each a difference in geometrical relationship for a main landmark between the values of the landmarks in the subset on a current frame and a corresponding subset on an onset frame;
the facial expression recognition unit to reduce the dimensionality of the descriptor by using primary component analysis (PCA) before using the descriptor in the model;
wherein at least one of:
multiple landmarks less than all landmarks of the face each have a subset of neighbor landmarks to use to determine the facial expression of the face;
each landmark of the face has a subset of neighbor landmarks used in the calculations;
wherein at least one of:
a selected number of closest nearby neighbor landmarks relative to a main landmark are included in the subset of the main landmark, and
any landmark within a certain distance from the main landmark is included in the subset of the main landmark;
wherein the membership of landmarks in the subsets remains the same for the same face on multiple frames;
the facial expression recognition unit to:
track landmarks on the image data;
normalize the image data forming the face by using reference face shape data to form an aligned face shape;
extract features comprising:
using the reference face shape data to determine the subsets, and
using the aligned face shape to determine the relationships; and
use a combination of the relationships in a discriminating label model to provide a face expression classification for the face;
wherein a plurality of differences between a current geometrical relationship and a corresponding geometrical relationship at a previous frame is placed into a latent dynamic continuous random field (LDCRF) model to match a facial expression classification to the face; and
wherein the plurality of differences are combined into a current descriptor, and the LDCRF model determines a facial expression classification for the current face by factoring the current descriptor and at least one descriptor of a previous frame with the same face.

### Claim 48

At least one computer readable article comprising a plurality of instructions that in response to being executed on a computing device, causes the computing device to:
obtain image data of at least one image that comprises content having at least one person's face;
determine at least one subset of landmarks for at least one main landmark, wherein each landmark has data to indicate information about a facial feature of the face;
determine a relationship among the at least one main landmark and the subset of landmarks; and
use a discriminative model to label a facial expression class for the face by using the relationship.

### Claim 49 (depends on 48)

The article of claim 48 wherein the relationship is a geometrical relationship determined by using linear regression-type calculations; and
the computing device caused to:
form a plurality of the subsets for a plurality of the main landmarks; and
determine the geometrical relationship for each of the subsets to form a group of the geometrical relationships; and
use the group of geometrical relationships to form a descriptor to use to determine a facial expression classification of the face.

### Claim 50 (depends on 48)

The article of claim 48 wherein the relationship is a geometrical relationship determined by using linear regression-type calculations;
the computing device caused to:
form a plurality of the subsets for a plurality of the main landmarks;
determine the geometrical relationship for each of the subsets to form a group of the geometrical relationships; and
use the group of geometrical relationships to form a descriptor to use to determine a facial expression classification of the face;
wherein each of the main landmarks of the plurality of main landmarks has a single subset as a main landmark while being available to be in other subsets as a neighbor landmark;
wherein the subsets overlap so that at least one landmark is in more than one subset;
the computing device caused to number the landmarks on the face to form an index, maintain the numbering while forming the relationships, and stack and concatenate the relationships in the same numerical sequence as the index number of the main landmark in order to form the descriptor;
wherein each geographical relationship is a vector of geographical relationship values wherein each value corresponds to a landmark in the subset, the computing device caused to use the geographical relationship values to form the descriptor by concatenating the geographical relationship values in each subset in a numerical order according to the index numbers of the landmarks corresponding to the geometrical relationship values, and within the space in the descriptor reserved for the subset by the index number of the main landmark;
wherein the descriptor is a vector that includes values that are each a difference in geometrical relationship for a main landmark between the values of the landmarks in the subset on a current frame and a corresponding subset on an onset frame;
the computing device caused to reduce the dimensionality of the descriptor by using primary component analysis (PCA) before using the descriptor in the model;
wherein at least one of:
multiple landmarks less than all landmarks of the face each have a subset of neighbor landmarks to use to determine the facial expression of the face;
each landmark of the face has a subset of neighbor landmarks used in the calculations;
wherein at least one of:
a selected number of closest nearby neighbor landmarks relative to a main landmark are included in the subset of the main landmark, and
any landmark within a certain distance from the main landmark is included in the subset of the main landmark;
wherein the membership of landmarks in the subsets remains the same for the same face on multiple frames;
the computing device caused to:
track landmarks on the image data;
normalize the image data forming the face by using reference face shape data to form an aligned face shape;
extract features comprising:
using the reference face shape data to determine the subsets, and
using the aligned face shape to determine the relationships; and
use a combination of the relationships in a discriminating label model to provide a face expression classification for the face;
wherein a plurality of differences between a current geometrical relationship and a corresponding geometrical relationship at a previous frame is placed into a latent dynamic continuous random field (LDCRF) model to match a facial expression classification to the face; and
wherein the plurality of differences are combined into a current descriptor, and the LDCRF model determines a facial expression classification for the current face by factoring the current descriptor and at least one descriptor of a previous frame with the same face.
