Systems and methods for slate optimization with recurrent neural networks

A computer-implemented method to train a machine-learned pointer network, the method comprising:

• obtaining, by one or more computing devices, data descriptive of the machine-learned pointer network configured to generate an output sequence from a list of candidate items, wherein the output sequence comprises a ranking of the candidate items, wherein the ranking of the candidate items orders the candidate items included in the output sequence based on a respective probability of user engagement with each candidate item included in the output sequence, wherein the machine-learned pointer network comprises an encoder network configured to receive the list of candidate items and provide an output that includes a sequence of latent memory states, a decoder network that operates over a plurality of decoding steps and is configured to receive a previously-selected candidate item for the output sequence and provide an output vector based at least in part on the previously-selected candidate item, and an attention network configured to receive the sequence of latent memory states and a query including the output vector from the decoder network, wherein the attention network is configured to produce a probability distribution associated with a next candidate item to include in the output sequence, and wherein, at each of the plurality of decoding steps, the attention network assigns scores to the list of candidate items as part of producing the probability distribution;
• training, by the one or more computing devices, the machine-learned pointer network based on a set of training data comprising a plurality of training examples, wherein each training example comprises a sequence of training items and a sequence of labels respectively associated with the sequence of training items, wherein the respective label for each training item indicates a level of user engagement with the training item,
• wherein training, by the one or more computing devices, the machine-learned pointer network comprises:
• determining, by the one or more computing devices, a per-step loss for two or more of the plurality of decoding steps, the per-step loss at each of the two or more of the plurality of decoding steps representing a performance evaluation of the machine-learned pointer network based on the set of training data at such decoding step, and wherein the per-step loss at each of the two or more of the plurality of decoding steps considers only training labels assigned to training items that have not been previously selected for inclusion in the output sequence;
• and
• modifying, by the one or more computing devices, one or more parameters of the machine-learned pointer network based at least in part on the per-step loss.

A computer-implemented method to generate an output sequence from a list of candidate items, the method comprising:

• ranking, by one or more computing devices, the list of candidate items to generate an initial sequence of candidate items;
• inputting, by the one or more computing devices, the initial sequence of candidate items into an encoder network configured to receive the initial sequence of candidate items and provide an output that includes a sequence of latent memory states;
• inputting, by the one or more computing devices, a previously-selected candidate item for the output sequence into a decoder network configured to receive the previously-selected candidate item and provide an output vector based at least in part on the previously-selected candidate item;
• and
• inputting the sequence of latent memory states and a query including the output vector from the decoder network into an attention network configured to receive the sequence of latent memory states and the query including the output vector from the decoder network, the attention network configured to produce a probability distribution associated with a next candidate item to include in the output sequence, wherein the output sequence comprises a ranking of the candidate items, wherein the ranking of the candidate items orders the candidate items included in the output sequence based on a respective probability of user engagement with each candidate item included in the output sequence;
• wherein the encoder network, the decoder network, and the attention network have been jointly trained on a training dataset comprising a plurality of training examples, wherein each training example comprises a sequence of training items and a sequence of labels respectively associated with the sequence of training items, wherein the respective label for each training item indicates a level of user engagement with the training item.