Meta-Q learning
AssigneeAmazon Technologies, Inc.
InventorsRasool Fakoor, Alexander Johannes Smola, Stefano Soatto, Pratik Anil Chaudhari
Life of this patent
Estimating the term…
Abstract
Techniques for Meta-Q-Learning (MQL) are described. A method of MQL may include receiving a request from an agent to perform adaptation based at least on task data associated with a new task collected by the agent, identifying a subset of meta-training data corresponding to the task data in a replay buffer, and adapting a policy using the subset of meta-training data and the task data to generate an adapted policy, wherein the adapted policy is used identify a next action for the agent to perform.
Claims20
A computer-implemented method comprising:
- receiving a set of training tasks;
- initializing a replay buffer and parameters for a policy;
- for each training task in the set of training tasks:
- obtaining training data for the training task by interacting with an environment;
- updating a context variable based on the training data;
- storing the training data in the replay buffer;
- and
- updating the parameters for the policy based on the training data to create a meta-trained policy;
- performing adaptation on the meta-trained policy using task data and a subset of the training data to generate an adapted meta-trained policy, wherein the subset of the training data is identified using the context variable and a propensity score, wherein the propensity score indicates a similarity between the task data and at least one training task from the set of training tasks.
A computer-implemented method comprising:
- performing, by an agent, adaptation based at least on task data associated with a new task collected by the agent;
- identifying, by the agent in a replay buffer using a propensity score, a subset of meta-training data corresponding to the task data, wherein the propensity score indicates a similarity between the new task and at least one training task from a set of training tasks;
- adapting a policy using the subset of meta-training data and the task data to generate an adapted policy;
- and
- identifying a next action for the agent to perform using the adapted policy.
A system comprising:
- a first one or more electronic devices to implement a storage service in a multi-tenant provider network;
- and
- a second one or more electronic devices to implement a Meta-Q-Learning (MQL) agent in the multi-tenant provider network, the MQL agent including instructions that upon execution by one or more processors cause the MQL agent to:
- perform adaptation based at least on task data associated with a new task collected by the MQL agent;
- identify, in a replay buffer of the storage service using a propensity score, a subset of meta-training data corresponding to the task data, wherein the propensity score indicates a similarity between the new task and at least one training task from a set of training tasks;
- adapt a policy using the subset of meta-training data and the task data to generate an adapted policy;
- and
- identify a next action for the MQL agent to perform using the adapted policy.
Dependencies are inferred from the claim text.