US 9,008,416 B2Granted

Collection of machine learning training data for expression recognition

AssigneeEmotient, Inc.

InventorsJavier Movellan, Marian Steward Bartlett, Ian Fasel, Gwen Ford Littlewort, Joshua Susskind, Jacob Whitehill

Life of this patent

Estimating the term…

Loading patent…

Abstract

Apparatus, methods, and articles of manufacture for implementing crowdsourcing pipelines that generate training examples for machine learning expression classifiers. Crowdsourcing providers actively generate images with expressions, according to cues or goals. The cues or goals may be to mimic an expression or appear in a certain way, or to “break” an existing expression recognizer. The images are collected and rated by same or different crowdsourcing providers, and the images that meet a first quality criterion are then vetted by expert(s). The vetted images are then used as positive or negative examples in training machine learning expression classifiers.

Claims20

Dependent claims

A computer-implemented method comprising steps of:

providing one or more cues to a first plurality of untrained providers for mimicking a predetermined expression;
receiving from the first plurality of untrained providers images created in response to the step of providing one or more cues;
sending requests to rate the images to a second plurality of untrained providers;
receiving ratings of the images from the second plurality of untrained providers, in response to the step of sending requests;
applying a first quality check to the images rated by the second plurality of untrained providers, the first quality check being based on the ratings of the second plurality of untrained providers, the step of applying the first quality check resulting in one or more images that passed the first quality check;
sending the one or more images that passed the first quality check to one or more experts, for rating by the one or more experts;
in response to the step of sending the one or more images that passed the first quality check, receiving one or more ratings from the one or more experts;
and
applying a second quality check to the images rated by the one or more experts, the second quality check being based on the one or more ratings of the one or more experts, the step of applying the second quality check resulting in one or more images that passed the second quality check.

6 more elements

A computer-implemented method comprising steps of:

providing one or more goals to a first plurality of untrained providers for breaking a classifier, the classifier being a machine-learning classifier of a predetermined expression of a predetermined emotion, affective state, or action unit associated with the predetermined expression;
receiving from the first plurality of untrained providers images created in response to the step of providing one or more goals, resulting in a plurality of received images;
checking the received images with the classifier and discarding images that do not meet a predetermined standard applied to output of the classifier, resulting in objectively qualified images that pass the predetermined standard;
sending to a second plurality of untrained providers requests to rate the objectively qualified images with respect to appearance of the predetermined emotion, affective state, or action unit in the objectively qualified images;
receiving ratings of the second plurality of untrained providers, in response to the requests;
applying a first quality check to the received images rated by the second plurality of untrained providers, the first quality check being based on the ratings of the second plurality of untrained providers, the step of applying the first quality check resulting in a plurality of images that passed the first quality check;
sending the plurality of images that passed the first quality check to one or more experts, for rating the images of the plurality of images that passed the first quality check by the one or more experts with respect to appearance of the predetermined emotion, affective state, or action unit in the images of the plurality of images that passed the first quality check;
receiving ratings from the one or more experts, in response to the step of sending the plurality of images that passed the first quality check;
and
applying a second quality check to the images rated by the one or more experts, the second quality check being based on the ratings of the one or more experts, the step of applying the second quality check resulting in one or more images that passed the second quality check.

7 more elements

A computer system comprising:

at least one processor;
at least memory device storing one or more cues for mimicking expression of a predetermined emotion, affective state, or action unit, the at least one memory device being coupled to the at least one processor;
and
at least one network interface coupled to the at least one processor and to at least one network;
wherein the at least one processor is configured to send through the at least one network interface the one or more cues to a first plurality of untrained providers;
to receive from the first plurality of untrained providers through the at least one network interface images created in response to the one or more cues;
to send to a second plurality of untrained providers through the at least one network interface requests to rate the images;
to receive through the at least one network interface ratings of the second plurality of untrained providers in response to the requests;
to apply a first quality check to the images rated by the second plurality of untrained providers, the first quality check being based on the ratings of the second plurality of untrained providers, thereby obtaining one or more images that passed the first quality check;
to send the one or more images that passed the first quality check to one or more experts, for rating by the one or more experts;
to receive one or more ratings from the one or more experts generated in response to the one or more images that passed the first quality check being sent to the one or more experts;
to apply a second quality check to the images rated by the one or more experts, the second quality check being based on the one or more ratings of the one or more experts, thereby obtaining one or more images that passed the second quality check;
and to train a classifier using the one or more images that passed the second quality check;
and
wherein the at least one processor is configured so that images that fail the first quality check are not sent to the one or more experts.

12 more elements

A computer system comprising:

at least one processor;
at least one memory device storing one or more goals for breaking a classifier of expression of a predetermined emotion, affective state, or action unit, the at least one memory device being coupled to the at least one processor;
and
at least one network interface coupled to the at least one processor and to at least one network;
wherein the at least one processor is configured to send through the at least one network interface the one or more goals to a first plurality of untrained providers;
to receive from the first plurality of untrained providers through the at least one network interface images created in response to the goals;
to check the received images with the classifier and to obtain objectively qualified images that meet a predetermined standard;
to send to a second plurality of untrained providers through the at least one network interface requests to rate the objectively qualified images with respect to appearance of expression of the predetermined emotion, affective state, or action unit in the objectively qualified images;
to receive through the at least one network interface ratings of the second plurality of untrained providers in response to the requests to rate the objectively qualified images;
to apply a first quality check to the objectively qualified images rated by the second plurality of untrained providers, the first quality check being based on the ratings of the second plurality of untrained providers, thereby obtaining one or more images that passed the first quality check;
to send the one or more images that passed the first quality check to one or more experts, for rating by the one or more experts with respect to appearance of expression of the predetermined emotion, affective state, or action unit in the one or more images that passed the first quality check;
to receive one or more ratings from the one or more experts generated in response to the one or more images that passed the first quality check being sent to the one or more experts;
to apply a second quality check to the images rated by the one or more experts, the second quality check being based on the one or more ratings of the one or more experts, thereby obtaining one or more images that passed the second quality check;
and to train a classifier using the one or more images that passed the second quality check as examples of appearance or non-appearance of expression of the predetermined emotion, affective state, or action unit;
and
wherein the at least one processor is configured so that the objectively qualified images that fail the first quality check are not sent to the one or more experts.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/177,174Filed 10 Feb 2014 · us provisional application
Priority: 8 Feb 2013
Published: US 2014/0321737 A130 Oct 2014
Granted: 14 Apr 2015
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G06V 10/776
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…
G06F 18/217
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: validation; Performance…
G06V 10/993
Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns:…
G06V 40/174
Human faces, e.g. facial parts, sketches or expressions: facial expression recognition