Systems and methods for orchestrating llm-augmented autonomous agents

A system for predicting an action by a plurality of language model augmented agents (LAAs), the system comprising:

• a memory that stores the plurality of LAAs and a plurality of processor executable instructions;
• a communication interface that receives a task instruction to be performed using an environment;
• and
• one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
• receiving, by a controller from the environment, an observation of a first state of the environment;
• selecting, by the controller, a LAA from the plurality of LAAs based on the task instruction and the observation;
• obtaining an output from the selected LAA generated using an input combining the task instruction, the observation, and an LAA-specific prompt template;
• determining, by the controller, the action based on the output;
• and
• causing the action to be performed on the environment thereby causing the first state of the environment to change to a second state.

A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:

• receiving, via a data interface, a task instruction to be performed using an environment;
• receiving, by a controller from the environment, an observation of a first state of the environment;
• selecting, by the controller, a language model augmented agents (LAA) from a plurality of LAAs based on the task instruction and the observation;
• obtaining an output from the selected LAA generated using an input combining the task instruction, the observation, and an LAA-specific prompt template;
• determining, by the controller, an action based on the output;
• and
• causing the action to be performed on the environment thereby causing the first state of the environment to change to a second state.