US 9,852,328 B2 · Emotion recognition in video conferencing

Abstract

Methods and systems for videoconferencing include recognition of emotions related to one videoconference participant such as a customer. This ultimately enables another videoconference participant, such as a service provider or supervisor, to handle angry, annoyed, or distressed customers. One example method includes the steps of receiving a video that includes a sequence of images, detecting at least one object of interest (e.g., a face), locating feature reference points of the at least one object of interest, aligning a virtual face mesh to the at least one object of interest based on the feature reference points, finding over the sequence of images at least one deformation of the virtual face mesh that reflect face mimics, determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions, and generating a communication bearing data associated with the facial emotion.

Claims20

Dependent claims

A computer-implemented method for video conferencing, the method comprising:

receiving a video including a sequence of images and an audio stream;
detecting at least one object of interest in one or more of the images;
locating feature reference points of the at least one object of interest;
aligning a virtual face mesh to the at least one object of interest in one or more of the images based at least in part on the feature reference points;
finding over the sequence of images at least one deformation of the virtual face mesh, wherein the at least one deformation is associated with at least one face mimic;
determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions;
recognizing a speech emotion in the audio stream of the at least one object of interest;
and
generating a communication bearing data associated with one or more of the facial emotion and the speech emotion.

6 more elements

A system, comprising:

one or more processors;
and
a non-transitory processor-readable medium coupled to the one or more processors, the non-transitory processor-readable medium comprising processor-executable instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving a video including a sequence of images and an audio stream;
detecting at least one object of interest in one or more of the images;
locating feature reference points of the at least one object of interest;
aligning a virtual face mesh to the at least one object of interest in one or more of the images based at least in part on the feature reference points;
finding over the sequence of images at least one deformation of the virtual face mesh, wherein the at least one deformation is associated with at least one face mimic;
determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions;
recognizing a speech emotion in the audio stream of the at least one object of interest;
and
generating a communication bearing data associated with one or more of the facial emotion and the speech emotion.

9 more elements

A non-transitory processor-readable medium comprising processor-executable instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving a video including a sequence of images and an audio stream;
detecting at least one object of interest in one or more of the images;
locating feature reference points of the at least one object of interest;
aligning a virtual face mesh to the at least one object of interest in one or more of the images based at least in part on the feature reference points;
finding over the sequence of images at least one deformation of the virtual face mesh, wherein the at least one deformation is associated with at least one face mimic;
determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions;
recognizing a speech emotion in the audio stream of the at least one object of interest;
and
generating a communication bearing data associated with one or more of the facial emotion and the speech emotion.

6 more elements

Dependencies are inferred from the claim text.

Emotion recognition in video conferencing

Life of this patent

Emotion recognition in video conferencing

Life of this patent

Abstract

Claims20