Chain-of-thought monitor for reward hack identification in generative response engines
AssigneeOpenAI OpCo, LLC
InventorBowen Baker
Life of this patent
Estimating the term…
Abstract
Systems and techniques are described herein for monitoring machine learning models for misbehavior. For example, an apparatus comprising one or more processors and configured to: provide, to a first machine learning model, an inner monologue from a second machine learning model associated with a user task, wherein the user task comprises natural language text; and obtain, from the first machine learning model, information indicative of misbehavior within the inner monologue. In some aspects, misbehavior is blocked during inference (e.g., runtime) and reward hacking behavior is penalized during training. Non-limiting examples of misbehavior include reward hacking, misgeneralization, sycophancy, or deception.
Claims20
A method of monitoring machine learning models for reward hacking, comprising:
- providing, to a second machine learning model, an inner monologue from a first machine learning model associated with a user task, wherein the user task comprises natural language text;
- and
- obtaining, from the second machine learning model, information indicative of misbehavior based on monitoring the inner monologue.
A computing device for monitoring machine learning models for reward hacking, comprising:
- at least one memory;
- and
- at least one processor coupled to the at least one memory and configured to:
- provide, to a second machine learning model, an inner monologue from a first machine learning model associated with a user task, wherein the user task comprises natural language text;
- and
- obtain, from the second machine learning model, information indicative of reward hacking behavior based on monitoring the inner monologue.
Dependencies are inferred from the claim text.