Back to results
US 12,755,209 B1Granted
Advantage weighted sampling for group-based relative policy optimization in machine learning
AssigneeIntuit Inc.
InventorsShai Ardazi, Ofir Ben Shoham, Shirli Di Castro Shashua, Gidi Zilbar
Life of this patent
Estimating the term…
Abstract
Aspects of the present disclosure provide techniques for optimizing a policy model in a reinforcement learning framework, involving generating output completions for an input query, evaluating the output completions with a reward model, and storing related data in a replay buffer. Certain aspects include computing weighted advantages for output completions sampled from the replay buffer and modifying policy model parameters based on these weighted advantages to enhance model performance.
Claims17
A method for optimizing a policy model in a reinforcement learning framework, comprising:
- generating a plurality of output completions for an input query using a first version of the policy model;
- evaluating the plurality of output completions using a reward model to assign scalar reward values;
- storing, in a replay buffer, one or more tuples comprising the input query, the output completions, the scalar reward values, and log probabilities of the plurality of output completions;
- sampling a group of stored tuples, including the one or more tuples, from the replay buffer;
- computing, based on the scalar reward values and the log probabilities, a weighted advantage for each output completion in the group of stored tuples relative to other output completions in the group of stored tuples using a normalized reward difference, wherein computing the weighted advantages for each output completion in the group comprises:
- computing a non-weighted advantage for each output completion in the group using the normalized reward difference;
- obtaining an absolute value of each of the non-weighted advantages;
- computing a sum of the absolute values of the non-weighted advantages;
- and
- computing the weighted advantages based on the sum of the absolute values of the non-weighted advantages;
- and
- automatically modifying one or more parameters of the policy model, based on the weighted advantages to produce a second version of the policy model, wherein the automatically modifying comprises:
- evaluating an objective function based on the weighted advantages;
- and
- performing, based on the evaluating, a gradient ascent step to modify the one or more parameters of the policy model to produce the second version of the policy model.
12 more elements
A system for optimizing a policy model in a reinforcement learning framework, comprising:
- one or more processors;
- a memory storing instructions that, when executed by the one or more processors, cause the system to:
- generate a plurality of output completions for an input query using a first version of the policy model;
- evaluate the plurality of output completions using a reward model to assign scalar reward values;
- store, in a replay buffer, one or more tuples comprising the input query, the output completions, the scalar reward values, and log probabilities of the plurality of output completions;
- sample a group of stored tuples, including the one or more tuples, from the replay buffer;
- compute, based on the scalar reward values and the log probabilities, a weighted advantage for each output completion in the group of stored tuples relative to other output completions in the group of stored tuples using a normalized reward difference, wherein to compute the weighted advantage for each output completion in the group the instructions cause the system:
- compute a non-weighted advantage for each output completion in the group using the normalized reward difference;
- obtain an absolute value of each of the non-weighted advantages;
- compute a sum of the absolute values of the non-weighted advantages;
- and
- compute the weighted advantages based on the sum of the absolute values of the non-weighted advantages;
- and
- automatically modify one or more parameters of the policy model, based on the weighted advantages to produce a second version of the policy model, wherein to automatically modify the one or more parameters of the policy model, the one or more processors are configured to:
- evaluate an objective function based on the weighted advantages;
- and
- perform, based on evaluating the objective function, a gradient ascent step to modify the one or more parameters of the policy model to produce the second version of the policy model.
14 more elements
A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
- generate a plurality of output completions for an input query using a first version of a policy model;
- evaluate the plurality of output completions using a reward model to assign scalar reward values;
- store, in a replay buffer, one or more tuples comprising the input query, the output completions, the scalar reward values, and log probabilities of the plurality of output completions;
- sample a group of stored tuples, including the one or more tuples, from the replay buffer;
- compute, based on the scalar reward values and the log probabilities, a weighted advantage for each output completion in the group of stored tuples relative to other output completions in the group of stored tuples using a normalized reward difference, wherein to compute the weighted advantage for each output completion in the group the instructions cause the system:
- compute a non-weighted advantage for each output completion in the group using the normalized reward difference;
- obtain an absolute value of each of the non-weighted advantages;
- compute a sum of the absolute values of the non-weighted advantages;
- and
- compute the weighted advantages based on the sum of the absolute values of the non-weighted advantages;
- and
- automatically modify one or more parameters of the policy model, based on the weighted advantages to produce a second version of the policy model, wherein to automatically modify the one or more parameters of the policy model, the one or more processors are configured to:
- evaluate an objective function based on the weighted advantages;
- and
- perform, based on evaluating the objective function, a gradient ascent step to modify the one or more parameters of the policy model to produce the second version of the policy model.
12 more elements
Dependencies are inferred from the claim text.