US 5,774,591 A · Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images

Abstract

A system tracks human head and facial features over time by analyzing a sequence of images. The system provides descriptions of motion of both head and facial features between two image frames. These descriptions of motion are further analyzed by the system to recognize facial movement and expression. The system analyzes motion between two images using parameterized models of image motion. Initially, a first image in a sequence of images is segmented into a face region and a plurality of facial feature regions. A planar model is used to recover motion parameters that estimate motion between the segmented face region in the first image and a second image in the sequence of images. The second image is warped or shifted back towards the first image using the estimated motion parameters of the planar model, in order to model the facial features relative to the first image. An affine model and an affine model with curvature are used to recover motion parameters that estimate the image motion between the segmented facial feature regions and the warped second image. The recovered motion parameters of the facial feature regions represent the relative motions of the facial features between the first image and the warped image. The face region in the second image is tracked using the recovered motion parameters of the face region. The facial feature regions in the second image are tracked using both the recovered motion parameters for the face region and the motion parameters for the facial feature regions. The parameters describing the motion of the face and facial features are filtered to derive mid-level predicates that define facial gestures occurring between the two images. These mid-level predicates are evaluated over time to determine facial expression and gestures occurring in the image sequence.

Claims31

Dependent claims

In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for recognizing facial expression of the human face, comprising the steps of:

defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs;
determining motion parameter values for each of the motion parameters of the parametric flow model defined by said defining step for each image pair in the sequence of image pairs;
identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression;
identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs;
and
associating the first onset of a facial expression with the first ending of a facial expression to identify a first facial expression formed during a first time period in the sequence of images.

3 more elements

A system for recognizing facial expression of the human face, the system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, comprising:

a motion estimation system for defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs, said motion estimation system determining motion parameter values for each of the motion parameters of the parametric flow model defined for each image pair in the sequence of image pairs;
and
a first detector for detecting a first facial expression, said first detector including:
means for identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression;
means for identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs;
and
means for associating the first onset of facial expression with the first ending of a facial expression to identify the first facial expression formed during a first time period in the sequence of images.

4 more elements

defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs;
determining motion parameter values for each of the motion parameters of the parametric flow model defined by said defining step for each image pair in the sequence of image pairs;
identifying a first series of image pairs in the sequence of image pairs with a first cyclical change in motion parameter values, the first cyclical change in motion parameter values indicating a movement in a first direction;
identifying a second series of image pairs in the sequence of image pairs with a second cyclical change in motion parameter values, the second cyclical change in motion parameter values indicating a movement in a second direction;
and
associating the first cyclical change in motion parameter values with the second cyclical change in motion parameter values to identity a first facial gesture formed during a first time period in the sequence of images.

3 more elements

In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for simultaneously recognizing facial expression and facial gestures, comprising the steps of:

defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs;
determining motion parameter values for each of the motion parameters of the parametric flow model defined by slid defining step for each image pair in the sequence of image pairs;
identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression;
identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs;
identifying a third series of image pairs in the sequence of image pairs with a first cyclical change in motion parameter values, the first cyclical change in motion parameter values indicating a movement in a first direction;
identifying a fourth series of image pairs in the sequence of image pairs with a second cyclical change in motion parameter values the second cyclical change in motion parameter values indicating a movement in a second direction;
associating the first cyclical change in motion parameter values with the second cyclical change in motion parameter values to identify a first facial gesture;
and
associating the first onset of a facial expression with the first ending of a facial expression to identify a first facial expression.

6 more elements

Dependencies are inferred from the claim text.

Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images

Life of this patent

Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images

Life of this patent

Abstract

Claims31