US 12,585,676 B1 · System and method for deep large language model thinking with gradient feedback alignment

Abstract

A large language model (LLM) is refined using an unsupervised alignment process for enhanced reasoning and deep thinking capabilities. An iterative search and optimization procedure is used to explore the space of possible thought generations enabling the model to improve deep thinking capabilities without direct supervision. The LLM is prompted to produce a first thought and response pair in response to a query. The thought and response pair is evaluated with a trained judge LLM, which provides feedback for the thought and for the response. A revised prompt is generated in response to the feedback and provided to the LLM, which produces second thought and response pair. The LLM is refined based on the first, non-preferred, thought and response pair and the second, preferred, thought and response pair to align with the preferred thought and response pair, e.g., using a direct preference optimization process.

Claims18

Dependent claims

A method for refining a large language model, comprising:

prompting the large language model to respond to a query with a thought and response pair, wherein the thought provides reasoning for the response, and, in response, receive a first thought and response pair;
evaluating the first thought and response pair in view of the query by a judge large language model that provides feedback based on scores for the thought and the response in the first thought and response pair;
generating a revised prompt based on the feedback to respond to the query with a thought and response pair;
providing the revised prompt to the large language model to receive a second thought and response pair;
and
refining the large language model based on the first thought and response pair and the second thought and response pair to modify operational parameters of the large language model by performing direct preference optimization with the first thought and response pair designated as a rejected response and the second thought and response pair designated as a chosen response.

3 more elements

A system for refining a large language model, comprising:

one or more processors;
and
a memory communicatively coupled with the one or more processors and storing instructions that, when executed by the one or more processors, causes the one or more processors to be configured to:
prompt the large language model to respond to a query with a thought and response pair, wherein the thought provides reasoning for the response, and in response receive a first thought and response pair;
evaluate the first thought and response pair in view of the query by a judge large language model that provides feedback based on scores for the thought and the response in the first thought and response pair;
generate a revised prompt based on the feedback to respond to the query with a thought and response pair;
provide the revised prompt to the large language model to receive a second thought and response pair;
and
refine the large language model based on the first thought and response pair and the second thought and response pair to modify operational parameters of the large language model by performing direct preference optimization with the first thought and response pair designated as a rejected response and the second thought and response pair designated as a chosen response.

6 more elements

Dependencies are inferred from the claim text.

System and method for deep large language model thinking with gradient feedback alignment

Life of this patent

System and method for deep large language model thinking with gradient feedback alignment

Life of this patent

Abstract

Claims18