US 11,451,742 B2Granted

Speech activity detection using dual sensory based learning

AssigneeBlackBerry Limited

InventorShiladitya Sircar

Life of this patent

Estimating the term…

Loading patent…

Abstract

A dual sensory input speech detection method includes receiving, at a first time, a first video image input of a conference participant of the video conference and a first audio input of the conference participant; communicating the first video image input to the video conference; identifying the first video image input as a first facial image of the conference participant; determining, based on the first facial image, the first video image input indicates the conference participant is in a speaking state; identifying the first audio input as a first speech sound; determining, while in the speaking state, the first speech sound originates from the conference participant; and communicating the first audio input to an audio output for the video conference.

Claims19

Dependent claims

A dual sensory input speech detection method in a video conference, the method comprising:

receiving, at a first time, a first video image input of a first conference participant of the video conference and a first audio input of the first conference participant, the first conference participant being one of a plurality of video conference participants;
communicating the first video image input to the video conference;
identifying the first video image input as a first facial image of the first conference participant;
determining, based on the first facial image, the first video image input indicates the first conference participant is in a speaking state;
identifying the first audio input as a first speech sound;
determining, while in the speaking state, the first speech sound originates from the first conference participant;
communicating the first audio input to an audio output for the video conference;
receiving, at a second time after the first time, a second audio input and a second video input from a second conference participant, the second conference participant being one of the plurality of video conference participants;
communicating the second audio input to the audio output for the video conference;
receiving, at a third time after the second time, a third video image input of the first conference participant and a third audio input of the first conference participant;
communicating the third video image input of the first conference participant to the video conference;
identifying the third video image input as a second facial image of the first conference participant;
determining, based on the second facial image, the third video image input does not indicate the first conference participant is in the speaking state;
determining the first conference participant is not in the speaking state;
and
adjusting the third audio input in response to determining that the first conference participant is not in the speaking state.

13 more elements

A communication device comprising:

a memory storing instructions;
a processor coupled to the memory and configured to execute the instructions to cause the communication device to:
receive, at a first time, a first video image input of a first conference participant of a video conference and a first audio input of the first conference participant, the first conference participant being one of a plurality of video conference participants;
communicate the first video image input to the video conference;
identify the first video image input as a first facial image of the first conference participant;
determine, based on the first facial image, the first video image input indicates the first conference participant is in a speaking state;
identifying the first audio input as a first speech sound;
determining, while in the speaking state, the first speech sound originates from the first conference participant;
communicating the first audio input to an audio output for the video conference;
receiving, at a second time after the first time, a second audio input and a second video input from a second conference participant, the second conference participant being one of the plurality of video conference participants;
communicating the second audio input to the audio output for the video conference;
receive, at a third time after the second time, a third video image input of the first conference participant and a third audio input of the first conference participant;
communicate the third video image input of the first conference participant to the video conference;
identify the third video image input as a second facial image of the first conference participant;
determine, based on the second facial image, the third video image input does not indicate the first conference participant is in the speaking state;
determine the first conference participant is not in the speaking state;
and
adjust the third audio input in response to determining that the first conference participant is not in the speaking state.

15 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/112,637Filed 4 Dec 2020
Published: US 2022/0182578 A19 Jun 2022
Granted: 20 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

H04N 7/15
Systems for two-way working: conference systems
G06V 10/751
Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g.…:…
G06V 20/40
Scenes; Scene-specific elements: in video content
G06V 40/165
Detection; Localisation; Normalisation: using facial parts and geometric relationships

Family

US 2022/0182578 A1Pre-grant publication · published 9 Jun 2022
US 11,451,742 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key