Back to results
US 2026/0088020 A1Published application
Customized keyword spotting using contextualized modeling
AssigneeMeta Platforms Technologies, LLC
InventorsTyler Vuong, Ming Sun, Zhaojun Yang
Life of this patent
Estimating the term…
Abstract
A method to train a contextualized custom keyword spotting model by adapting a contextualized automatic speech framework with modified training labels. The target training labels may include the words in the ground-truth transcript that also appear in the input bias text and are then used to train the model with loss determination. This approach enables the building of customized keyword spotting models without additional alignment or word-segmented data.
Claims20
A method for training a customized keyword spotting model, comprising:
- receiving an input audio signal;
- receiving a transcript that corresponds with the input audio signal;
- receiving an input bias text;
- modifying the transcript to include words that overlap with the input bias text to generate a modified transcript;
- generating audio embeddings based on processing the input audio signal through an audio encoder;
- generating text embeddings based on processing the input bias text through a text encoder;
- combining the audio embeddings and text embeddings using a text biasing layer to generate combined embeddings;
- providing predicted probabilities for respective tokens based on the combined embeddings;
- determining a loss between the predicted probabilities and the modified transcript using Connectionist Temporal Classification (CTC) loss;
- and
- updating parameters of the model based on the loss.
8 more elements
A method for keyword spotting comprising:
- receiving an input audio signal;
- receiving a target keyword as input bias text;
- generating audio embeddings based on processing the input audio signal through an audio encoder;
- generating text embeddings based on processing the target keyword through a text encoder;
- combining the audio embeddings and text embeddings using a text biasing layer to generate combined embeddings;
- providing predicted probabilities for each token of one or more tokens based on the combined embeddings;
- determining whether the target keyword is present in the input audio signal based on the predicted probabilities;
- and
- transmitting an alert based on the determining that the target keyword is present.
6 more elements
A device comprising:
- a processor;
- and
- a memory storing instructions that, when executed by the processor, cause the device to:
- receive an input audio signal;
- determine the use of a target keyword in the input audio signal based on a customized keyword spotting model associated with spotting one or more keywords in the input audio signal, wherein the customized keyword spotting model comprises:
- receiving the input audio signal;
- receiving the target keyword as input bias text;
- generating audio embeddings based on processing the input audio signal through an audio encoder;
- generating text embeddings based on processing the target keyword through a text encoder;
- combining the audio embeddings and text embeddings using a text biasing layer to generate combined embeddings;
- providing predicted probabilities for each token of one or more tokens based on the combined embeddings;
- determining whether the target keyword is present in the input audio signal based on the predicted probabilities;
- and
- transmitting a message based on the determining that the target keyword is present;
- and
- send instructions to execute an action based on the use of the target keyword.
13 more elements
Dependencies are inferred from the claim text.