Back to results
US 12,608,622 B1Granted
Replay buffer integration for group-based relative policy optimization in machine learning
AssigneeIntuit Inc.
InventorsShirli Dicastro, Shai Ardazi, Ofir Ben Shoham, Gidi Zilbar
Life of this patent
Estimating the term…
Abstract
Aspects of the present disclosure provide techniques for optimizing a policy model in a reinforcement learning framework, involving generating output completions for an input query, evaluating the output completions with a reward model, and storing related data in a replay buffer. Certain aspects include computing advantages for output completions sampled from the replay buffer and modifying policy model parameters based on these advantages to enhance model performance.
Claims15
A method for optimizing a policy model in a reinforcement learning framework, comprising:
- performing an instance of a generation phase based on a first schedule configured by a first scheduler that controls scheduling of instances of the generation phase, wherein the first schedule results in the instances of the generation phase being initiated at a first frequency, wherein the first scheduler triggers the instance of the generation phase based on determining an amount of data that has been sampled from a replay buffer, and wherein the performing of the instance comprises:
- determining an input query from a task distribution;
- generating a plurality of output completions for the input query using a first version of the policy model;
- evaluating the plurality of output completions using a reward model to assign scalar reward values;
- and
- storing, in the replay buffer, one or more tuples comprising the input query, the output completions, the scalar reward values, and log probabilities of the plurality of output completions;
- and
- performing a respective instance of an optimization phase based on a second schedule configured by a second scheduler that controls scheduling of respective instances of the optimization phase, wherein the second schedule results in the respective instances of the optimization phase being initiated at a second frequency that is higher than the first frequency, wherein at least two of the instances of the optimization phase involve sampling a same stored tuple from the replay buffer to form at least two different groups used in the at least two of the instances of the optimization phase, and wherein the performing of the respective instance comprises:
- sampling a group of stored tuples, including the one or more tuples, from the replay buffer;
- computing, based on the scalar reward values and the log probabilities, an advantage for each output completion in the group of stored tuples relative to other output completions in the group of stored tuples using normalized reward differences;
- and
- automatically modifying one or more parameters of the policy model, based on the computed advantages to produce a second version of the policy model.
9 more elements
A system for optimizing a policy model in a reinforcement learning framework, comprising:
- one or more processors;
- a memory storing instructions that, when executed by the one or more processors, cause the system to:
- perform an instance of a generation phase based on a first schedule configured by a first scheduler that controls scheduling of instances of the generation phase, wherein the first scheduler triggers the instance of the generation phase based on determining an amount of data that has been sampled from a replay buffer, wherein the first schedule results in the instances of the generation phase being initiated at a first frequency, and wherein the performing of the instance comprises:
- determining an input query from a task distribution;
- generating a plurality of output completions for the input query using a first version of the policy model;
- evaluating the plurality of output completions using a reward model to assign scalar reward values;
- and
- storing, in the replay buffer, one or more tuples comprising the input query, the output completions, the scalar reward values, and log probabilities of the plurality of output completions;
- and
- perform a respective instance of an optimization phase based on a second schedule configured by a second scheduler that controls scheduling of respective instances of the optimization phase, wherein the second schedule results in the respective instances of the optimization phase being initiated at a second frequency that is higher than the first frequency, wherein at least two of the instances of the optimization phase involve sampling a same stored tuple from the replay buffer to form at least two different groups used in the at least two of the instances of the optimization phase, and wherein the performing of the respective instance comprises:
- sampling a group of stored tuples, including the one or more tuples, from the replay buffer;
- computing, based on the scalar reward values and the log probabilities, an advantage for each output completion in the group of stored tuples relative to other output completions in the group of stored tuples using normalized reward differences;
- and
- modifying one or more parameters of the policy model, based on the computed advantages to produce a second version of the policy model.
11 more elements
A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
- perform an instance of a generation phase based on a first schedule configured by a first scheduler that controls scheduling of instances of the generation phase, wherein the first scheduler triggers the instance of the generation phase based on determining an amount of data that has been sampled from a replay buffer, wherein the first schedule results in the instances of the generation phase being initiated at a first frequency, and wherein the performing of the instance comprises:
- determining an input query from a task distribution;
- generating a plurality of output completions for the input query using a first version of a policy model;
- evaluating the plurality of output completions using a reward model to assign scalar reward values;
- and
- storing, in the replay buffer, one or more tuples comprising the input query, the output completions, the scalar reward values, and log probabilities of the plurality of output completions;
- and
- perform a respective instance of an optimization phase based on a second schedule configured by a second scheduler that controls scheduling of respective instances of the optimization phase, wherein the second schedule results in the respective instances of the optimization phase being initiated at a second frequency that is higher than the first frequency, wherein at least two of the instances of the optimization phase involve sampling a same stored tuple from the replay buffer to form at least two different groups used in the at least two of the instances of the optimization phase, and wherein the performing of the respective instance comprises:
- sampling a group of stored tuples, including the one or more tuples, from the replay buffer;
- computing, based on the scalar reward values and the log probabilities, an advantage for each output completion in the group of stored tuples relative to other output completions in the group of stored tuples using normalized reward differences;
- and
- modifying one or more parameters of the policy model, based on the computed advantages to produce a second version of the policy model.
9 more elements
Dependencies are inferred from the claim text.