US 2023/0068798 A1Published application

Active speaker detection using image data

AssigneeAmazon Technologies, Inc.

InventorsTyler Jerel Etchart, Vivek Yadav, Pradeep Natarajan

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system can operate a speech-controlled device to perform active speaker detection to detect an utterance using image data showing a user speaking the utterance. This enables the device to perform utterance detection using the image data and/or determine which user is speaking the utterance. To perform active speaker detection, the device processes the image data to determine expression parameters associated with the user's face and generates facial measurements based on the expression parameters. For example, the device can use the expression parameters to generate a 3D model including an agnostic facial representation and determine a mouth aspect ratio by measuring a mouth height and a mouth width of the agnostic facial representation. As the mouth aspect ratio changes when the user is speaking, the device can determine that the user is speaking and/or detect an utterance based on an amount of variation of the mouth aspect ratio.

Claims20

Dependent claims

A computer-implemented method, the method comprising:

receiving first image data;
determining that a first face is represented in a first portion of the first image data;
inputting the first portion of the first image data to a trained model to determine first data representing at least one first parameter corresponding to a first facial expression;
using the first data to generate second data including a first agnostic facial representation having the first facial expression, the first agnostic facial representation having uniform identity and uniform pose;
using the second data to determine third data representing a first width of a first mouth in the first agnostic facial representation;
using the second data to determine fourth data representing a first height of the first mouth;
using the third data and the fourth data to determine a portion of fifth data representing a first ratio between the first height of the first mouth and the first width of the first mouth, the fifth data representing a first series of ratio values;
determining a first standard deviation value using the first series of ratio values represented in the fifth data;
determining that the first standard deviation value exceeds a threshold value;
and
in response to determining that the first standard deviation value exceeds the threshold value, determining that a user associated with the first face is speaking.

8 more elements

A computer-implemented method, the method comprising:

receiving first image data;
determining that a first face is represented in the first image data;
processing the first image data to determine first data representing at least one first parameter corresponding to a first facial expression;
using the first data to generate second data that includes a portion of a first agnostic facial representation representing a first mouth;
using the second data to determine a portion of third data representing a first ratio value between a first mouth height of the first mouth and a first mouth width of the first mouth, the third data including a first plurality of ratio values;
determining fourth data representing a first amount of variation in the first plurality of ratio values;
and
using the fourth data to determine that the first face is speaking.

5 more elements

A system comprising:

at least one processor;
and
memory including instructions operable to be executed by the at least one processor to cause the system to:
receive first image data;
determine that a first face is represented in the first image data;
process the first image data to determine first data representing at least one first parameter corresponding to a first facial expression;
use the first data to generate second data that includes a portion of a first agnostic facial representation representing a first mouth;
use the second data to determine a portion of third data representing a first ratio value between a first mouth height of the first mouth and a first mouth width of the first mouth, the third data including a first plurality of ratio values;
determine fourth data representing a first amount of variation in the first plurality of ratio values;
and
use the fourth data to determine that the first face is speaking.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/465,143Filed 2 Sep 2021
Published: 2 Mar 2023
Term ends: Not estimatedThe source records don’t support an estimate.

Classification24 codes

G06T 7/50
Image analysis: depth or shape recovery
G10L 15/25
Speech recognition using non-acoustical features: using position of the lips, movement of the lips or face analysis
G06T 7/251
Using feature-based methods, e.g. the tracking of corners or segments: involving models
G06T 7/277
Analysis of motion: involving stochastic approaches, e.g. using Kalman filters

Family

No related US applications are recorded in the indexed filing history.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key