Back to results
US 11,893,348 B2Granted
Training a machine learning system for keyword prediction with neural likelihood
AssigneeRoyal Bank of Canada
InventorsHareesh Pallikara Bahuleyan, Layla El Asri
Life of this patent
Estimating the term…
Abstract
Computer implemented methods and systems are provided for generating diverse key phrases while maintaining competitive output quality. A system for training a sequence to sequence (S2S) machine learning model is proposed where neural unlikelihood objective approaches are used at (1) a target token level to discourage the generation of repeating tokens, and (2) a copy token level to avoid copying repetitive tokens from the source text. K-step ahead token prediction approaches are also proposed as an additional mechanism to augment the approach to further enhance the overall diversity of key phrase outputs.
Claims20
A system for training a sequence to sequence (S2S) machine learning model for predicting keywords, the system comprising:
- at least one computer memory having stored thereon the S2S machine learning model, the S2S machine learning model comprising a plurality of parameters representative of a decoder, the decoder including a generation data model architecture and a copy data model architecture;
- at least one processor, in communication with the at least one computer memory, configured to:
- receive a first data set comprising a plurality of source token sets and related ground truth token sets;
- extract a second data set of target vocabulary tokens from the first data set comprising a subset of source tokens and related ground truth tokens of the first data set;
- train the S2S machine learning model for predicting keywords by, for each source token in a first source token set:
- processing, with the decoder, a first source token set encoder representation and a respective source token encoder representation to generate a predicted keyword, wherein processing with the decoder comprises:
- processing the first source token set encoder representation and a previous ground truth token embedding to generate a hidden state;
- generating a first set keyword probability distribution of the copy data model architecture based on normalizing, over the source tokens in the first source token set, an attention mechanism interrelation value between the respective source token encoder representation and the hidden state;
- generating a second set keyword probability distribution of the generation data model architecture based on normalizing, over the target vocabulary tokens, the attention mechanism interrelation value;
- determining a probability of generating a keyword from the second data set based on a vocabulary token parameter processing the hidden state, a related ground truth token, and the vocabulary normalized attention mechanism interrelation value;
- generating a probability of generating the keyword from the first source token set based on the probability of generating the keyword from the second data set;
- and
- generating a predicted keyword based on applying the probability of generating the keyword from the second data set to the second set keyword probability distribution and applying the probability of generating the keyword from the first token source set to the first set keyword probability distribution;
- updating the plurality of parameters by:
- determining a generation loss based on comparing the predicted keyword to a first exclusion list of ground truth tokens;
- determining a copy loss based on comparing the predicted keyword to a second exclusion list of source tokens and ground truth tokens;
- and
- adjusting the plurality of parameters based on the copy loss, the generation loss, and a comparison of the predicted keyword and a respective predicted keyword ground truth token to penalize the decoder for generating repetitive keywords;
- and
- store the trained S2S machine learning model for predicting keywords in the at least one computer memory.
17 more elements
A system for training a sequence to sequence (S2S) machine learning model for predicting keywords, the system comprising:
- at least one computer memory having stored thereon the S2S machine learning model, the S2S machine learning model comprising a plurality of parameters representative of a decoder, the decoder including a generation data model architecture and a copy data model architecture;
- at least one processor, in communication with the at least one computer memory, configured to:
- receive a first data set comprising a plurality of source token sets, and related ground truth token sets;
- extract a second data set of target vocabulary tokens from the first data set comprising a subset of source tokens and ground truth tokens of the first data set;
- train the S2S machine learning model for predicting keywords by, for each source token in a first source token set:
- generating a predicted keyword based on a first source token set keyword probability distribution of the copy data model architecture associated with a probability of generating the keyword from the first source token set, based on the respective source token, and a second set keyword probability distribution of the generation data model architecture, based on the respective source token, and associated with the probability of generating the keyword from the second data set;
- updating the plurality of parameters by:
- determining a generation loss based on comparing the predicted keyword to a first exclusion list of ground truth tokens;
- determining a copy loss based on comparing the predicted keyword to a second exclusion list of source tokens and ground truth tokens;
- and
- adjusting the plurality of parameters based on the copy loss, the generation loss, and a comparison of the predicted keyword and a respective predicted keyword ground truth token to penalize the decoder for generating repetitive keywords;
- and
- store the trained S2S machine learning model in the at least one computer memory.
10 more elements
A method for training a sequence to sequence (S2S) machine learning model for predicting keywords, the S2S machine learning model comprising a plurality of parameters representative of a decoder, the decoder including a generation data model architecture and a copy data model architecture;
- the method comprising:
- receiving a first data set comprising a plurality of source token sets, and related ground truth token sets;
- extracting a second data set of target vocabulary tokens from the first data set comprising a subset of source tokens and ground truth tokens of the first data set;
- training the S2S machine learning model by, for each source token in a first source token set:
- generating a predicted keyword based on a first source token set keyword probability distribution of the copy data model architecture associated with a probability of generating the keyword from the first source token set, based on the respective source token, and a second set keyword probability distribution of the generation data model architecture, based on the respective source token, and associated with the probability of generating the keyword from the second data set;
- updating the plurality of parameters by:
- determining a generation loss based on comparing the predicted keyword to a first exclusion list of ground truth tokens;
- determining a copy loss based on comparing the predicted keyword to a second exclusion list of source tokens and ground truth tokens;
- and
- adjusting the plurality of parameters based on the copy loss, the generation loss, and a comparison of the predicted keyword and a respective predicted keyword ground truth token to penalize the decoder for generating repetitive keywords;
- and
- storing the trained S2S machine learning model.
9 more elements
A non-transitory computer readable storage medium having stored therein computer executable program code, which when execited by the processor, cases the processor to:
- receive a first data set comprising a plurality of source token sets, and related ground truth token sets;
- extract a second data set of target vocabulary tokens from the first data set comprising a subset of source tokens and ground truth tokens of the first data set;
- train an S2S machine learning model for predicting keywords, the S2S machine learning model comprising a plurality of parameters representative of a decoder, the decoder including a generation data model architecture and a copy data model architecture:
- by, for each source token in a first source token set:
- generating a predicted keyword based on a first source token set keyword probability distribution of the copy data model architecture associated with a probability of generating the keyword from the first source token set, based on the respective source token, and a second set keyword probability distribution of the generation data model architecture, based on the respective source token, and associated with the probability of generating the keyword from the second data set;
- updating the plurality of parameters by:
- determining a generation loss based on comparing the predicted keyword to a first exclusion list of ground truth tokens;
- determining a copy loss based on comparing the predicted keyword to a second exclusion list of source tokens and ground truth tokens;
- adjusting the plurality of parameters based on the copy loss, the generation loss, and a comparison of the predicted keyword and a respective predicted keyword ground truth token to penalize the decoder for generating repetitive keywords;
- and
- store the trained S2S machine learning model in at least one computer memory.
8 more elements
Dependencies are inferred from the claim text.