Back to results
US 10,176,802 B1Granted
Lattice encoding using recurrent neural networks
AssigneeAmazon Technologies, Inc.
InventorsFaisal Ladhak, Ankur Gandhe, Markus Dreyer, Ariya Rastrow, Björn Hoffmeister, Lambert Mathias
Life of this patent
Estimating the term…
Abstract
An automatic speech recognition (ASR) system may convert an ASR output lattice into a matrix form, thus maintaining certain information included in the lattice that might otherwise be lost in an N-best list output. The matrix representation of the lattice may be encoded using a recurrent neural network (RNN) to create a vector representation of the lattice. The vector representation may then be used by the system to perform additional operations, such as ASR results confirmation.
Claims20
A computer-implemented method comprising:
- receiving audio data corresponding to an utterance;
- performing automatic speech recognition (ASR) processing on the audio data to obtain an ASR lattice portion;
- determining a matrix representing the ASR lattice portion, wherein the matrix comprises at least:
- a first representation of a first path of the ASR lattice portion, the first representation comprising:
- first data indicating that a first node is connected to a second node, the first data being associated with a first word, and
- second data indicating that the second node is connected to a third node, the second data being associated with a second word;
- and
- a second representation of a second path of the ASR lattice portion, the second representation comprising:
- third data indicating that the first node is connected to the third node, the third data being associated with a third word;
- encoding the matrix using a recurrent neural network (RNN) encoder to obtain an encoded vector corresponding to the ASR lattice portion, the encoding comprising:
- processing the first data using the RNN encoder to create a first encoded vector, wherein the first encoded vector represents a portion of the first path up to the second node,
- processing the second data and the first encoded vector using the RNN encoder to create a second encoded vector, wherein the second encoded vector represents the first path,
- processing the third data using the RNN encoder to create a third encoded vector, wherein the third encoded vector represents the second path, and
- combining the second encoded vector and the third encoded vector to create a fourth encoded vector, wherein the fourth encoded vector represents the ASR lattice portion;
- and
- determine at least one ASR hypothesis based at least in part on the fourth encoded vector.
13 more elements
A computing device comprising:
- at least one processor;
- and
- at least one memory including instructions operable to be executed by the at least one processor to configure the device to:
- receive first audio data corresponding to a first utterance;
- process at least a portion of the first audio data to determine a matrix representing at least a portion of an automatic speech recognition (ASR) lattice, the matrix comprising at least:
- first data representing a first arc of the ASR lattice, the first arc originating at a first node of the ASR lattice and terminating at a second node of the ASR lattice, and
- second data representing a second arc of the ASR lattice, the second arc originating at a third node of the ASR lattice and terminating at the second node of the ASR lattice;
- process the first data using a recurrent neural network (RNN) encoder to create a first encoded vector;
- process the second data using the RNN encoder to create a second encoded vector;
- combine the first encoded vector and the second encoded vector to create a third encoded vector;
- and
- determine at least one ASR hypothesis based at least in part on the third encoded vector.
9 more elements
A computer-implemented method comprising:
- receiving first audio data corresponding to a first utterance;
- processing at least a portion of the first audio data to determine a matrix representing at least a portion of an automatic speech recognition (ASR) lattice, the matrix comprising at least:
- first data representing a first arc of the ASR lattice, the first arc originating at a first node of the ASR lattice and terminating at a second node of the ASR lattice, and
- second data representing a second arc of the ASR lattice, the second arc originating at a third node of the ASR lattice and terminating at the second node of the ASR lattice;
- processing the first data using a recurrent neural network (RNN) encoder to create a first encoded vector;
- processing the second data using the RNN encoder to create a second encoded vector;
- combining the first encoded vector and the second encoded vector to create a third encoded vector;
- and
- determining at least one ASR hypothesis based at least in part on the third encoded vector.
6 more elements
Dependencies are inferred from the claim text.