US 12,730,969 B2Granted

Multistage alignment for generating artificial intelligence training data

AssigneeDeepgram, Inc.

InventorsJeffrey Michael Ward, Andrew Nathan Seagraves, Matthew Benson Lightman, Scott Ivan Stephenson, Adam Joseph Sypniewski

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Artificial intelligence (AI) training data includes pairs of known input and output. Training an AI model includes generating an output from the model, comparing it against a known output, and modifying the model parameters until the model generates an output close to the known output. For speech recognition AI models, the training data includes pairs of audio files and corresponding transcripts. Dividing the audio files into chunks can be beneficial for the training of a speech recognition AI model. However, transcripts may not include timing or alignment data to find a corresponding transcript portion when dividing an audio file. A first stage alignment can generate timing labels for the speech tokens in a transcript. When dividing an audio file into audio file chunks, the timing labels can be used to find corresponding transcript portions to audio file chunks. A second stage alignment can improve the accuracy of the timing labels.

Claims20

Dependent claims

A method comprising:

receiving an audio file and an associated transcript of the audio file;
generating ground truth tokens from the transcript;
converting the audio file into audio features at a plurality of timestamps of the audio file;
generating predicted tokens for each timestamp;
performing a first stage alignment of the ground truth tokens and the predicted tokens, generating first stage timings, wherein the first stage timings comprise the timestamps of the matched predicted tokens assigned to the matched ground truth tokens;
wherein performing the first stage alignment further comprises:
generating a probability density function (PDF) for each timestamp, wherein the PDF comprises a matrix of two dimensions comprising number of time stamps and a vocabulary size;
generating a predicted sequence of tokens from the PDF, by identifying the most probable token at each time stamp;
generating the first stage timings, based at least in part on the predicted sequence of tokens;
performing a second stage alignment comprising:
generating a plurality of second stage alignment paths, each path comprising an alignment of the ground truth tokens to the predicted tokens;
generating a score for each alignment path, based at least in part on the first stage timings,
wherein generating the scores, based at least in part on the first stage timings, further comprises:
upweighting the probability of rarer tokens in the PDF, wherein less frequent tokens are scored higher than more frequent tokens;
selecting a final alignment path based at least in part on the scores;
generating second stage timings, based at least in part on the selected final alignment path;
assigning timing labels to portions of the transcript, at least in part, based on the second stage timings;
generating input/output training data, based at least in part on the assigned timing labels, the input/output training data, comprising the portions of the transcript and corresponding portions of the audio file, wherein the correspondence is determined at least in part based on the assigned timing labels;
training a supervised artificial intelligence network with the generated training data;
and
performing inference operations comprising:
receiving an audio file without a known transcript;
and
with the trained supervised artificial intelligence network, generating a transcript of the audio file without a known transcript.

21 more elements

A non-transitory computer storage medium that stores executable program instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

receiving an audio file and an associated transcript of the audio file;
generating ground truth tokens from the transcript;
converting the audio file into audio features at a plurality of timestamps of the audio file;
generating predicted tokens for each timestamp;
performing a first stage alignment of the ground truth tokens and the predicted tokens, generating first stage timings, wherein the first stage timings comprise the timestamps of the matched predicted tokens assigned to the matched ground truth tokens;
wherein performing the first stage alignment further comprises:
generating a probability density function (PDF) for each timestamp, wherein the PDF comprises a matrix of two dimensions comprising number of time stamps and a vocabulary size;
generating a predicted sequence of tokens from the PDF, by identifying the most probable token at each time stamp;
generating the first stage timings, based at least in part on the predicted sequence of tokens;
performing a second stage alignment comprising:
generating a plurality of second stage alignment paths, each path comprising an alignment of the ground truth tokens to the predicted tokens;
generating a score for each alignment path, based at least in part on the first stage timings,
wherein generating the scores, based at least in part on the first stage timings, further comprises:
upweighting the probability of rarer tokens in the PDF, wherein less frequent tokens are scored higher than more frequent tokens;
selecting a final alignment path based at least in part on the scores;
generating second stage timings, based at least in part on the selected final alignment path;
assigning timing labels to portions of the transcript, at least in part, based on the second stage timings;
generating input/output training data, based at least in part on the assigned timing labels, the input/output training data, comprising the portions of the transcript and corresponding portions of the audio file, wherein the correspondence is determined at least in part based on the assigned timing labels;
training a supervised artificial intelligence network with the generated training data;
and
performing inference operations comprising:
receiving an audio file without a known transcript;
and
with the trained supervised artificial intelligence network, generating a transcript of the audio file without a known transcript.

21 more elements

A system comprising one or more processors, wherein the one or more processors are configured to perform operations comprising:

receiving an audio file and an associated transcript of the audio file;
generating ground truth tokens from the transcript;
converting the audio file into audio features at a plurality of timestamps of the audio file;
generating predicted tokens for each timestamp;
performing a first stage alignment of the ground truth tokens and the predicted tokens, generating first stage timings, wherein the first stage timings comprise the timestamps of the matched predicted tokens assigned to the matched ground truth tokens;
wherein performing the first stage alignment further comprises:
generating a probability density function (PDF) for each timestamp, wherein the PDF comprises a matrix of two dimensions comprising number of time stamps and a vocabulary size;
generating a predicted sequence of tokens from the PDF, by identifying the most probable token at each time stamp;
generating the first stage timings, based at least in part on the predicted sequence of tokens;
performing a second stage alignment comprising:
generating a plurality of second stage alignment paths, each path comprising an alignment of the ground truth tokens to the predicted tokens;
generating a score for each alignment path, based at least in part on the first stage timings,
wherein generating the scores, based at least in part on the first stage timings, further comprises:
upweighting the probability of rarer tokens in the PDF, wherein less frequent tokens are scored higher than more frequent tokens;
selecting a final alignment path based at least in part on the scores;
generating second stage timings, based at least in part on the selected final alignment path;
assigning timing labels to portions of the transcript, at least in part, based on the second stage timings;
generating input/output training data, based at least in part on the assigned timing labels, the input/output training data, comprising the portions of the transcript and corresponding portions of the audio file, wherein the correspondence is determined at least in part based on the assigned timing labels;
training a supervised artificial intelligence network with the generated training data;
and
performing inference operations comprising:
receiving an audio file without a known transcript;
and
with the trained supervised artificial intelligence network, generating a transcript of the audio file without a known transcript.

21 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/598,217Filed 7 Mar 2024 · continuation
Priority: 6 Mar 2024
Published: US 2025/0284889 A111 Sep 2025
Granted: 8 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

G06F 40/284
Recognition of textual entities: lexical analysis, e.g. tokenisation or collocates
G10L 13/08
Speech synthesis; Text to speech systems: text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme…
G10L 15/02
Speech recognition: feature extraction for speech recognition; Selection of recognition unit
G10L 15/04
Speech recognition: segmentation; Word boundary detection