Back to results
US 12,744,045 B2Granted
Using machine learning and discrete tokens to estimate different sound sources from audio mixtures
AssigneeGoogle Inc.
InventorsHakan Erdogan, Scott Thomas Wisdom, John Hershey, Zalán Borsos, Marco Tagliasacchi, Neil Zeghidour, Xuankai Chang
Life of this patent
Estimating the term…
Abstract
A system and method are disclosed. Audio input comprising the mixed audio signals is received by one or more client devices. The audio input is converted into a plurality of discrete tokens. A plurality of sound sources, each corresponding to a subset of discrete tokens of a plurality of subsets of discrete tokens, is determined using a trained machine learning model.
Claims19
A method comprising:
- receiving audio input comprising mixed audio signals provided by one or more client devices of one or more users of a video conference platform that participate in a video conference;
- converting the audio input into a plurality of discrete tokens comprising one or more of a plurality of acoustic tokens, which capture at least sound events or acoustic characteristics within the mixed audio signals, or a plurality of semantic tokens, which are derived from text data extracted from the mixed audio signals and capture at least associated meanings and contextual representations of words or subword units, wherein converting the audio input into the plurality of discrete tokens comprises:
- providing, to a first machine learning model, input comprising the audio input;
- and
- obtaining, from the first machine learning model, one or more outputs identifying the plurality of discrete tokens;
- and
- determining, using a second machine learning model, a plurality of sound sources each corresponding to a subset of discrete tokens of a plurality of subsets of discrete tokens, wherein a transcript of at least one of the plurality of sounds sources is generated for the video conference.
4 more elements
A system comprising:
- a memory device;
- and
- a processing device coupled to the memory device, the processing device to perform operations comprising:
- receiving audio input comprising mixed audio signals provided by one or more client devices of one or more users of a video conference platform that participate in a video conference;
- converting the audio input into a plurality of discrete tokens comprising one or more of a plurality of acoustic tokens, which capture at least sound events or acoustic characteristics within the mixed audio signals, or a plurality of semantic tokens, which are derived from text data extracted from the mixed audio signals and capture at least associated meanings and contextual representations of words or subword units, wherein converting the audio input into the plurality of discrete tokens comprises:
- providing, to a first machine learning model, input comprising the audio input;
- and
- obtaining, from the first machine learning model, one or more outputs identifying the plurality of discrete tokens;
- and
- determining, using a second machine learning model, a plurality of sound sources each corresponding to a subset of discrete tokens of a plurality of subsets of discrete tokens, wherein a transcript of at least one of the plurality of sounds sources is generated for the video conference.
7 more elements
A non-transitory computer readable medium comprising instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
- receiving audio input comprising mixed audio signals provided by one or more client devices of one or more users of a video conference platform that participate in a video conference;
- converting the audio input into a plurality of discrete tokens comprising one or more of a plurality of acoustic tokens, which capture at least sound events or acoustic characteristics within the mixed audio signals, or a plurality of semantic tokens, which are derived from text data extracted from the mixed audio signals and capture at least associated meanings and contextual representations of words or subword units, wherein converting the audio input into the plurality of discrete tokens comprises:
- providing, to a first machine learning model, input comprising the audio input;
- and
- obtaining, from the first machine learning model, one or more outputs identifying the plurality of discrete tokens;
- and
- determining, using a second machine learning model, a plurality of sound sources each corresponding to a subset of discrete tokens of a plurality of subsets of discrete tokens, wherein a transcript of at least one of the plurality of sounds sources is generated for the video conference.
4 more elements
Dependencies are inferred from the claim text.