Knowledge distillation in multi-arm bandit, neural network models for real-time online optimization

A system comprising:

• one or more servers of a knowledge distillation system, wherein the one or more servers include:
• one or more processors;
• and
• one or more non-transitory, computer-readable storage media storing code that, when executed by the one or more processors, cause the one or more processors to:
• pretrain, using a contextual multi-arm bandit (MAB) algorithm, a teacher model neural network to determine sets of contextual MAB data tuples based on sets of input context vectors;
• train, using the pretrained teacher model neural network, a student model neural network to determine sets of contextual MAB data tuples based on sets of input context vectors;
• determine, by the trained student model neural network using the contextual MAB algorithm, contextual MAB data tuples for a context vector;
• store the contextual MAB data tuples in a replay buffer that enables online updates of the trained student model neural network using batches of data in the replay buffer, wherein the replay buffer is in a same memory as the teacher model neural network and the student model neural network;
• sample, by the pretrained teacher model neural network, the contextual MAB data tuples from the replay buffer;
• determine, by the pretrained teacher model neural network using the contextual MAB algorithm, augmented contextual data MAB data tuples for the context vector;
• correlate, by the pretrained teacher model neural network, the sampled contextual MAB data tuples with the augmented contextual data MAB data tuples;
• store the correlated augmented contextual MAB data tuples in the replay buffer;
• and
• further train, by the trained student model neural network accessing the replay buffer, the trained student model neural network using a batch of the correlated augmented contextual MAB data within the replay buffer, wherein:
• each contextual MAB data tuple comprises a data set (xi, ai, ri), wherein “i” represents any index value,
• each augmented contextual MAB data tuple comprises a data set (xi, au, {circumflex over (r)}l), wherein “u” represents any index value, and
• zi represents an ith context vector, ai represents a counterfactual ith bandit arm, ri represents an award correlated to the context vector xi and the bandit arm ai, au represents a uth bandit arm, and {circumflex over (r)}l represents a softened reward.

A non-transitory computer-readable storage medium storing computer program code, wherein the computer program code when executed by a processor performs operations comprising:

• pretraining, using a contextual multi-arm bandit (MAB) algorithm, a teacher model neural network of a knowledge distillation system to determine sets contextual multi-arm bandit (MAB) data tuples based on sets of input context vectors;
• training, using the pretrained teacher model neural network, a student model neural network of the knowledge distillation system to determine sets of contextual MAB data tuples based on sets of input context vectors;
• determining, by the trained student model neural network using the contextual MAB algorithm, contextual MAB data tuples for a context vector;
• storing the contextual MAB data tuples in a replay buffer of the knowledge distillation system, wherein the replay buffer is in a same memory as the teacher model neural network and the student model neural network and enables online updates of the trained student model neural network using batches of data in the replay buffer;
• sampling, by the pretrained teacher model neural network, the contextual MAB data tuples from the replay buffer;
• determining, by the pretrained teacher model neural network using the contextual MAB algorithm, augmented contextual data MAB data tuples for the context vector;
• correlating, by the pretrained teacher model neural network, the sampled contextual MAB data tuples with the augmented contextual data MAB data tuples;
• storing the correlated augmented contextual MAB data tuples in the replay buffer;
• further training, by the trained student model neural network accessing the replay buffer, the trained student model neural network using a batch of the augmented contextual data of contextual MAB data tuples within the replay buffer;
• and
• determining, by the further trained student model neural network processing user features obtained from a user device, a creative to present to the user, wherein:
• each contextual MAB data tuple comprises a data set (xi, ai, ri), wherein “i” represents any index value,
• each augmented contextual MAB data tuple comprises a data set (xi, au, {circumflex over (r)}l), wherein “u” represents any index value, and
• xi represents an it context vector, ai represents a counterfactual ith bandit arm, ri represents an award correlated to the context vector xi and the bandit arm ai, au represents a uth bandit arm, and {circumflex over (r)}l represents a softened reward.