US 12,585,658 B1Granted

Generating augmented output sequences by a neural network using external databases

AssigneeGoogle Inc.

InventorTerrance Paul McCartney, Jr.

Life of this patent

Estimating the term…

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating augmented output sequences by using a language model neural network and a plurality of databases. Each database is associated with a respective level of confidence. Each database stores a respective plurality of pre-generated embeddings in an embedding space.

Claims20

Dependent claims

A computer-implemented method comprising:

maintaining a plurality of databases that are each associated with a respective level of confidence, wherein each database stores a respective plurality of pre-generated embeddings in an embedding space that have been generated from source subsequences;
generating an initial output sequence by using a language model neural network, the initial output sequence comprising one or more subsequences;
for each of the one or more subsequences, determining a confidence score that indicates a level of confidence of information represented by the subsequence, wherein determining the confidence score comprises:
generating, based on the subsequence included in the initial output sequence generated by using the language model neural network, a query embedding in the embedding space;
searching through the plurality of databases to identify, from the respective pluralities of pre-generated embeddings stored in the plurality of databases, one or more similar pre-generated text embeddings, wherein the query embedding satisfies a distance criterion with each of the one or more similar pre-generated text embeddings;
and
computing the confidence score for the subsequence based on the levels of confidence associated with one or more of the plurality of databases that store the one or more similar pre-generated subsequences;
identifying, from the one or more subsequences and based on the confidence scores determined for the one or more subsequences, a particular subsequence that has a particular confidence score, wherein the particular confidence score is determined based on a particular level of confidence associated with a particular database storing one or more similar pre-generated text embeddings that satisfy the distance criterion with a query embedding of the particular subsequence;
and
in response to identifying the particular subsequence, generating, based on the initial output sequence of the language model neural network, an augmented output sequence that modifies the initial output sequence to remove the particular subsequence initially generated by the language model neural network.

7 more elements

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

maintaining a plurality of databases that are each associated with a respective level of confidence, wherein each database stores a respective plurality of pre-generated embeddings in an embedding space that have been generated from source subsequences;
generating an initial output sequence by using a language model neural network, the initial output sequence comprising one or more subsequences;
for each of the one or more subsequences, determining a confidence score that indicates a level of confidence of information represented by the subsequence, wherein determining the confidence score comprises:
generating, based on the subsequence included in the initial output sequence generated by using the language model neural network, a query embedding in the embedding space;
searching through the plurality of databases to identify, from the respective pluralities of pre-generated embeddings stored in the plurality of databases, one or more similar pre-generated text embeddings, wherein the query embedding satisfies a distance criterion with each of the one or more similar pre-generated text embeddings;
and
computing the confidence score for the subsequence based on the levels of confidence associated with one or more of the plurality of databases that store the one or more similar pre-generated subsequences;
identifying, from the one or more subsequences and based on the confidence scores determined for the one or more subsequences, a particular subsequence that has a particular confidence score, wherein the particular confidence score is determined based on a particular level of confidence associated with a particular database storing one or more similar pre-generated text embeddings that satisfy the distance criterion with a query embedding of the particular subsequence;
and
in response to identifying the particular subsequence, generating, based on the initial output sequence of the language model neural network, an augmented output sequence that modifies the initial output sequence to remove the particular subsequence initially generated by the language model neural network.

7 more elements

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

maintaining a plurality of databases that are each associated with a respective level of confidence, wherein each database stores a respective plurality of pre-generated embeddings in an embedding space that have been generated from source subsequences;
generating an initial output sequence by using a language model neural network, the initial output sequence comprising one or more subsequences;
for each of the one or more subsequences, determining a confidence score that indicates a level of confidence of information represented by the subsequence, wherein determining the confidence score comprises:
generating, based on the subsequence included in the initial output sequence generated by using the language model neural network, a query embedding in the embedding space;
searching through the plurality of databases to identify, from the respective pluralities of pre-generated embeddings stored in the plurality of databases, one or more similar pre-generated text embeddings, wherein the query embedding satisfies a distance criterion with each of the one or more similar pre-generated text embeddings;
and
computing the confidence score for the subsequence based on the levels of confidence associated with one or more of the plurality of databases that store the one or more similar pre-generated subsequences;
identifying, from the one or more subsequences and based on the confidence scores determined for the one or more subsequences, a particular subsequence that has a particular confidence score, wherein the particular confidence score is determined based on a particular level of confidence associated with a particular database storing one or more similar pre-generated text embeddings that satisfy the distance criterion with a query embedding of the particular subsequence;
and
in response to identifying the particular subsequence, generating, based on the initial output sequence of the language model neural network, an augmented output sequence that modifies the initial output sequence to remove the particular subsequence initially generated by the language model neural network.

7 more elements

Dependencies are inferred from the claim text.

patentgrep

Loading patent…

Abstract

A computer-implemented method comprising:

maintaining a plurality of databases that are each associated with a respective level of confidence, wherein each database stores a respective plurality of pre-generated embeddings in an embedding space that have been generated from source subsequences;
generating an initial output sequence by using a language model neural network, the initial output sequence comprising one or more subsequences;
for each of the one or more subsequences, determining a confidence score that indicates a level of confidence of information represented by the subsequence, wherein determining the confidence score comprises:
generating, based on the subsequence included in the initial output sequence generated by using the language model neural network, a query embedding in the embedding space;
searching through the plurality of databases to identify, from the respective pluralities of pre-generated embeddings stored in the plurality of databases, one or more similar pre-generated text embeddings, wherein the query embedding satisfies a distance criterion with each of the one or more similar pre-generated text embeddings;
and
computing the confidence score for the subsequence based on the levels of confidence associated with one or more of the plurality of databases that store the one or more similar pre-generated subsequences;
identifying, from the one or more subsequences and based on the confidence scores determined for the one or more subsequences, a particular subsequence that has a particular confidence score, wherein the particular confidence score is determined based on a particular level of confidence associated with a particular database storing one or more similar pre-generated text embeddings that satisfy the distance criterion with a query embedding of the particular subsequence;
and
in response to identifying the particular subsequence, generating, based on the initial output sequence of the language model neural network, an augmented output sequence that modifies the initial output sequence to remove the particular subsequence initially generated by the language model neural network.

7 more elements