Back to results
US 11,232,796 B2Granted
Voice activity detection using audio and visual analysis
AssigneeMeta Platforms, Inc.
InventorsJun Yang, Joshua Bingham
Life of this patent
Estimating the term…
Abstract
A method of detecting voice activity includes performing a video analysis on a frame of video signal to determine a position of a user in the frame and to identify one or more beams of a corresponding audio signal associated with a region including the position of the user. The identified one or more beams of audio signal are analyzed to determine whether voice is present in the frame. When a user is not identified during the video analysis of the frame of video signal, audio analysis is not performed on the corresponding frame of audio signal.
Claims18
A method comprising:
- receiving, from a client device including an image capturing device and an audio recording device, a video signal of an environment within a field-of-view of the image capturing device and an audio signal of the environment, the audio signal comprising a plurality of audio beams each associated with a different region of the environment;
- detecting a subject of interest in a frame of the received video signal;
- determining a position of the subject of interest in the frame;
- for a first audio beam of the plurality of audio beams, determining whether a first region corresponding to the first audio beam includes the position of the subject of interest;
- responsive to determining that the first region includes the position of the subject of interest, determining whether voice is present in the first audio beam;
- for a second audio beam of the plurality of audio beams, determining whether a second region corresponding to the second audio beam includes the position of the subject of interest;
- responsive to determining that the second region does not include the position of the subject of interest, not determining whether voice is present in the second audio beam;
- and
- outputting a voice activity detection signal responsive to detecting that voice is present.
6 more elements
A non-transitory computer-readable medium comprising computer program instructions that when executed by a computer processor of an online system cause the processor to perform steps comprising:
- receiving, from a client device including an image capturing device and an audio recording device, a video signal of an environment within a field-of-view of the image capturing device and an audio signal of the environment, the audio signal comprising a plurality of audio beams each associated with a different region of the environment;
- detecting a subject of interest in a frame of the received video signal;
- determining a position of the subject of interest in the frame;
- for a first audio beam of the plurality of audio beams, determining whether a first region corresponding to the first audio beam includes the position of the subject of interest;
- responsive to determining that the first region includes the position of the subject of interest, determining whether voice is present in the first audio beam;
- for a second audio beam of the plurality of audio beams, determining whether a second region corresponding to the second audio beam includes the position of the subject of interest;
- responsive to determining that the second region does not include the position of the subject of interest, not determining whether voice is present in the second audio beam;
- and
- outputting a voice activity detection signal responsive to detecting that voice is present.
6 more elements
A system comprising:
- a processor;
- and
- a non-transitory computer-readable medium comprising computer program instructions that when executed by the processor of an online system causes the processor to perform steps comprising:
- receiving, from a client device including an image capturing device and an audio recording device, a video signal of an environment within a field-of-view of the image capturing device and an audio signal of the environment, the audio signal comprising a plurality of audio beams each associated with a different region of the environment;
- detecting a subject of interest in a frame of the received video signal;
- determining a position of the subject of interest in the frame;
- for a first audio beam of the plurality of audio beams, determining whether a first region corresponding to the first audio beam includes the position of the subject of interest;
- responsive to determining that the first region includes the position of the subject of interest, determining whether voice is present in the first audio beam;
- and
- for a second audio beam of the plurality of audio beams, determining whether a second region corresponding to the second audio beam includes the position of the subject of interest;
- responsive to determining that the second region does not include the position of the subject of interest, not determining whether voice is present in the second audio beam;
- and
- outputting a voice activity detection signal responsive to detecting that voice is present.
10 more elements
Dependencies are inferred from the claim text.