Back to results
US 12,762,440 B2Granted
Machine learning techniques for denoising input sequences
AssigneeOptum, Inc.
InventorsAyan Sengupta, Amit Kumar, Suman Roy, Sourabh Kumar Bhattacharjee
Life of this patent
Estimating the term…
Abstract
Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing data denoising. Certain embodiments of the present invention utilize systems, methods, and computer program products that perform data denoising by utilizing at least one of encoder transformer machine learning models, decoder transformer machine learning models, contextual relevance determination non-linear machine learning models, contextual relevance decision-making machine learning models, denoising decision-making machine learning model, and denoising decision gates.
Claims20
A computer-implemented method comprising:
- determining, by one or more processors, an input data object corresponding to an input token;
- determining, by the one or more processors and using an encoder transformer machine learning model, a hidden representation of the input token;
- determining, by the one or more processors and using a contextual relevance determination non-linear machine learning model, and based at least in part on the hidden representation, a contextual relevance probability, associated with the input token, that indicates a likelihood that the input token provides reliable contextual information usable in determining denoised representations of one or more surrounding input tokens;
- determining, by the one or more processors and using the contextual relevance decision-making machine learning model, a contextual relevance representation corresponding to the input token based at least in part on the contextual relevance probability and the hidden representation, wherein the contextual relevance representation comprises a copy probability, a removal probability, or a generation probability, and the contextual relevance representation describes the hidden representation or a masked representation based on a contextual relevance probability threshold;
- determining, by the one or more processors and using a decoder transformer machine learning model, a denoised representation for the input token based at least in part on the contextual relevance representation and a preceding denoised representation for a preceding input token that precedes the input token in accordance with a token order;
- determining, by the one or more processors, a denoised sequence based at least in part on the denoised representation;
- and
- initiating, by the one or more processors, a performance of one or more prediction-based actions based at least in part on the denoised sequence.
5 more elements
A system comprising:
- one or more processors;
- and
- one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
- determining an input data object corresponding to an input token;
- determining, using an encoder transformer machine learning model, a hidden representation of the input token;
- determining, using a contextual relevance determination non-linear machine learning model, and based at least in part on the hidden representation, a contextual relevance probability associated with the input token, that indicates a likelihood that the input token provides reliable contextual information usable in determining denoised representations of one or more surrounding input tokens;
- determining, using a contextual relevance decision-making machine learning model, a contextual relevance representation corresponding to the input token based at least in part on the contextual relevance probability and the hidden representation, wherein the contextual relevance representation comprises a copy probability, a removal probability, or a generation probability, and the contextual relevance representation describes the hidden representation or a masked representation based on a contextual relevance probability threshold;
- determining, using a decoder transformer machine learning model, a denoised representation for the input token based at least in part on the contextual relevance representation and a preceding denoised representation for a preceding input token that precedes the input token in accordance with a token order;
- determining a denoised sequence based at least in part on the denoised representation;
- and
- initiating a performance of one or more prediction-based actions based at least in part on the denoised sequence.
8 more elements
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- determining an input data object corresponding to an input token;
- determining, using an encoder transformer machine learning model, a hidden representation of the input token;
- determining, using a contextual relevance determination non-linear machine learning model, and based at least in part on the hidden representation, a contextual relevance probability associated with the input token, that indicates a likelihood that the input token provides reliable contextual information usable in determining denoised representations of one or more surrounding input tokens;
- determining, using a contextual relevance decision-making machine learning model, a contextual relevance representation corresponding to the input token based at least in part on the contextual relevance probability and the hidden representation, wherein the contextual relevance representation comprises a copy probability, a removal probability, or a generation probability, and the contextual relevance representation describes the hidden representation or a masked representation based on a contextual relevance probability threshold;
- determining, using a decoder transformer machine learning model, a denoised representation for the input token based at least in part on the contextual relevance representation and a preceding denoised representation for a preceding input token that precedes the input token in accordance with a token order;
- determining a denoised sequence based at least in part on the denoised representation;
- and
- initiating a performance of one or more prediction-based actions based at least in part on the denoised sequence.
5 more elements
Dependencies are inferred from the claim text.