Methods and systems for selecting actions from a set of actions to be performed in an environment affected by delays

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for selecting at least one action from a plurality of actions to be performed in an environment, the operations comprising:

• maintaining, for each action from the plurality of actions, count data indicative of a number of times the action has been performed and a difference between the number of times the action has been performed and a number of observed resulting rewards for the action, each reward being a numeric value that measures an outcome of a given action;
• determining, from the count data and a bandit score provided by a bandit model, an expected score for each action from the plurality of actions, the bandit score provided by the bandit model for a given history of performed actions and observed rewards, and the expected score determined by determining an expected value of the bandit score given a likelihood of some of the plurality of actions having unobserved pending rewards;
• and
• selecting, from the plurality of actions and based on the expected score for each action, the at least one action to be performed in the environment.