Determining action selection policies of an execution device

A system for generating an action selection policy for completing a task in an environment that includes an execution device and one or more other devices, the system comprising:

• one or more processors;
• and
• one or more computer-readable memories coupled to the one or more processors and having instructions stored thereon that are executable by the one or more processors to perform operations comprising:
• at a current iteration of a plurality of iterations,
• obtaining an action selection policy in the current iteration, wherein the action selection policy specifies a respective probability of selecting an action among a plurality of possible actions in a current state of the execution device, wherein the current state results from a previous action taken by the execution device in a previous state, and each action of the plurality of possible actions leads to a respective next state if performed by the execution device when the execution device is in the current state;
• obtaining a respective first reward for each action of the plurality of possible actions in the current state, wherein the respective first reward for the action represents a gain attributed to the action towards completing the task;
• computing a first reward for the current state based on respective first rewards for the plurality of possible actions and the action selection policy in the current iteration, wherein the first reward for the current state represents a gain attributed to the current state towards completing the task;
• computing a respective regret value of each action of the plurality of possible actions based on a difference between the respective first reward for the action and the first reward for the current state;
• computing an incremental action selection policy based on the respective regret value of the action in the current iteration but not any regret value of the action in any iteration prior to the current iteration, wherein the incremental action selection policy is used for determining an action selection policy in a next iteration;
• computing a second reward for the current state based on the respective first rewards for the plurality of possible actions and the incremental action selection policy, wherein the second reward for the current state includes partial information of the action selection policy in the next iteration;
• and
• determining the action selection policy in the next iteration based on the second reward of the current state;
• and
• controlling actions of the execution device according to the action selection policy.

A non-transitory, computer-readable storage medium storing one or more instructions executable by a computer system to perform operations for generating an action selection policy for completing a task in an environment that includes an execution device and one or more other devices, the operations comprising:

• at a current iteration of a plurality of iterations,
• obtaining an action selection policy in the current iteration, wherein the action selection policy specifies a respective probability of selecting an action among a plurality of possible actions in a current state of the execution device, wherein the current state results from a previous action taken by the execution device in a previous state, and each action of the plurality of possible actions leads to a respective next state if performed by the execution device when the execution device is in the current state;
• obtaining a respective first reward for each action of the plurality of possible actions in the current state, wherein the respective first reward for the action represents a gain attributed to the action towards completing the task;
• computing a first reward for the current state based on respective first rewards for the plurality of possible actions and the action selection policy in the current iteration, wherein the first reward for the current state represents a gain attributed to the current state towards completing the task;
• computing a respective regret value of each action of the plurality of possible actions based on a difference between the respective first reward for the action and the first reward for the current state;
• computing an incremental action selection policy based on the respective regret value of the action in the current iteration but not any regret value of the action in any iteration prior to the current iteration, wherein the incremental action selection policy is used for determining an action selection policy in a next iteration;
• computing a second reward for the current state based on the respective first rewards for the plurality of possible actions and the incremental action selection policy, wherein the second reward for the current state includes partial information of the action selection policy in the next iteration;
• and
• determining the action selection policy in the next iteration based on the second reward of the current state;
• and
• controlling actions of the execution device according to the action selection policy.