# US 11,451,742 B2 · Speech activity detection using dual sensory based learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,742 B2 - Document ID: US11451742B2 - Status: Granted - Kind: B2 - Application number: 17/112,637 - Filed: 2020-12-04 - Published: 2022-06-09 - Granted: 2022-09-20 - Earliest claimed: 2020-12-04 - Assignees: BlackBerry Limited - Applicants: BlackBerry Limited - Inventors: Shiladitya Sircar - Web page: https://patentgrep.com/patent/US11451742B2 ## CPC - H04N 7/15: Conference systems - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 40/165: {using facial parts and geometric relationships} - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)} - H04M 3/569: {using the instant speaker's algorithm (speech detection per se G10L25/78)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract A dual sensory input speech detection method includes receiving, at a first time, a first video image input of a conference participant of the video conference and a first audio input of the conference participant; communicating the first video image input to the video conference; identifying the first video image input as a first facial image of the conference participant; determining, based on the first facial image, the first video image input indicates the conference participant is in a speaking state; identifying the first audio input as a first speech sound; determining, while in the speaking state, the first speech sound originates from the conference participant; and communicating the first audio input to an audio output for the video conference. ## Claims ### Claim 1 A dual sensory input speech detection method in a video conference, the method comprising: receiving, at a first time, a first video image input of a first conference participant of the video conference and a first audio input of the first conference participant, the first conference participant being one of a plurality of video conference participants; communicating the first video image input to the video conference; identifying the first video image input as a first facial image of the first conference participant; determining, based on the first facial image, the first video image input indicates the first conference participant is in a speaking state; identifying the first audio input as a first speech sound; determining, while in the speaking state, the first speech sound originates from the first conference participant; communicating the first audio input to an audio output for the video conference; receiving, at a second time after the first time, a second audio input and a second video input from a second conference participant, the second conference participant being one of the plurality of video conference participants; communicating the second audio input to the audio output for the video conference; receiving, at a third time after the second time, a third video image input of the first conference participant and a third audio input of the first conference participant; communicating the third video image input of the first conference participant to the video conference; identifying the third video image input as a second facial image of the first conference participant; determining, based on the second facial image, the third video image input does not indicate the first conference participant is in the speaking state; determining the first conference participant is not in the speaking state; and adjusting the third audio input in response to determining that the first conference participant is not in the speaking state. ### Claim 2 (depends on 1) The method of claim 1, wherein the audio output is transmitted to other conference participants. ### Claim 3 (depends on 1) The method of claim 1, wherein the first and second facial images include a lip image of the first conference participant. ### Claim 4 (depends on 1) The method of claim 1, further comprising prior to determining the first conference participant is not in the speaking state, identifying the third audio input as not a speech sound of the first conference participant. ### Claim 5 (depends on 1) The method of claim 1, wherein adjusting the third audio input comprises not communicating the third audio input to the audio output of the video conference. ### Claim 6 (depends on 1) The method of claim 1, wherein adjusting the third audio input comprises attenuating the third audio input and communicating an attenuated audio input to the audio output of the video conference. ### Claim 7 (depends on 1) The method of claim 1, wherein the method is performed by a dual sensory input speech detection system, and wherein the method further comprises training the dual sensory input speech detection system. ### Claim 8 (depends on 7) The method of claim 7, wherein training the dual sensory input speech detection system comprises one or more of identifying lip landmarks of the first conference participant, or calculating a mouth ratio of the first conference participant. ### Claim 9 (depends on 8) The method of claim 8, wherein training the dual sensory input speech detection system further comprises identifying speech characteristics of the first conference participant, wherein identifying the first audio input as the first speech sound comprises comparing the first audio input to the speech characteristics of the first conference participant, and wherein determining the first conference participant is in the speaking state comprises comparing one or more of the lip landmarks or the mouth ratio with the first video image input. ### Claim 10 (depends on 1) The method of claim 1 further comprising: receiving, at a fourth time, a fourth video image input of the second conference participant and a fourth audio input of the second conference participant; communicating the fourth video image input of the second conference participant to the video conference; identifying the fourth video image input as a fourth facial image of the second conference participant; determining, based on the fourth facial image, the fourth video image input does not indicate the second conference participant is in the speaking state; determining the second conference participant is not in the speaking state; and adjusting the fourth audio input in response to determining that the second conference participant is not in the speaking state. ### Claim 11 A communication device comprising: a memory storing instructions; a processor coupled to the memory and configured to execute the instructions to cause the communication device to: receive, at a first time, a first video image input of a first conference participant of a video conference and a first audio input of the first conference participant, the first conference participant being one of a plurality of video conference participants; communicate the first video image input to the video conference; identify the first video image input as a first facial image of the first conference participant; determine, based on the first facial image, the first video image input indicates the first conference participant is in a speaking state; identifying the first audio input as a first speech sound; determining, while in the speaking state, the first speech sound originates from the first conference participant; communicating the first audio input to an audio output for the video conference; receiving, at a second time after the first time, a second audio input and a second video input from a second conference participant, the second conference participant being one of the plurality of video conference participants; communicating the second audio input to the audio output for the video conference; receive, at a third time after the second time, a third video image input of the first conference participant and a third audio input of the first conference participant; communicate the third video image input of the first conference participant to the video conference; identify the third video image input as a second facial image of the first conference participant; determine, based on the second facial image, the third video image input does not indicate the first conference participant is in the speaking state; determine the first conference participant is not in the speaking state; and adjust the third audio input in response to determining that the first conference participant is not in the speaking state. ### Claim 12 (depends on 11) The communication device of claim 11, wherein the audio output is transmitted to other conference participants. ### Claim 13 (depends on 11) The communication device of claim 11, wherein the first facial image comprises a lip image of the first conference participant. ### Claim 14 (depends on 11) The communication device of claim 11, wherein the instructions further cause the communication device to identify the third audio input as not a speech sound of the first conference participant. ### Claim 15 (depends on 11) The communication device of claim 11, wherein the instructions further cause the communication device to not communicate the third audio input to the audio output of the video conference. ### Claim 16 (depends on 11) The communication device of claim 11, wherein the instructions further cause the communication device to attenuate the third audio input and communicate an attenuated audio input to the audio output of the video conference. ### Claim 17 (depends on 11) The communication device of claim 11, further comprising a dual sensory input speech detection system, and wherein the instructions further cause the communication device to train the dual sensory input speech detection system. ### Claim 18 (depends on 17) The communication device of claim 17, wherein the instructions further cause the communication device to identify lip landmarks of the first conference participant, or calculate a mouth ratio of the first conference participant. ### Claim 19 (depends on 18) The communication device of claim 18, wherein the instructions further cause the communication device to: identify speech characteristics of the first conference participant; compare the first audio input to the speech characteristics of the first conference participant; and compare one or more of the lip landmarks or the mouth ratio with the first video image input.