US 2026/0037873 A1 · COMBINING SEARCH, SELF-CRITIQUE AND REINFORCEMENT LEARNING FOR AUTONOMOUS WEB AGENTS

Abstract

Systems and methods include improved planning and reasoning capabilities of a web agent, which interacts with a real world website. The agent improves with autonomous experience and requires limited supervision. An MCTS-based search routine is deployed over web pages, a base model is used as an initial distribution, and a number of rationales and possible web actions are explored. AI feedback and self-criticism are used, and the model provides self-evaluation at each node.

Claims20

Dependent claims

A method for generating an optimized model policy for one or more agents, the method comprising:

(a) receiving an initial model policy, a replay buffer of trajectories, and an instruction;
(b) determining a number of the trajectories for each instruction using the initial model policy;
(c) constructing preference pairs at each node in each of the trajectories;
(d) adding the preference pairs to the replay buffer;
(e) updating the initial model policy creating an updated initial model policy based on the replay buffer, an executed plan of steps associated with the instruction executed in the updated initial model policy, a history of the executed plan and context of the executed plan, and the initial model policy;
and
(f) repeating steps (b)-(e) for a number of iterations to generate the optimized model policy based on the updated initial model policy.

4 more elements

A computer system for generating an optimized model policy for one or more agents, the computer system comprising:

a hardware processor;
and
a non-volatile storage medium storing instructions that when executed by the hardware processor perform operations comprising:
(a) receiving an initial model policy, a replay buffer of trajectories, and an instruction;
(b) determining a number of the trajectories for each instruction using the initial model policy;
(c) constructing preference pairs at each node in each of the trajectories;
(d) adding the preference pairs to the replay buffer;
(e) updating the initial model policy creating an updated initial model policy based on the replay buffer, an executed plan of steps associated with the instruction executed in the updated initial model policy, a history of the executed plan and context of the executed plan, and the initial model policy;
and
(f) repeating steps (b)-(e) for a number of iterations to generate the optimized model policy based on the updated initial model policy.

7 more elements

A computer program product for generating an optimized model policy for one or more agents, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to perform operations comprising:

(a) receiving an initial model policy, a replay buffer of trajectories, and an instruction;
(b) determining a number of the trajectories for each instruction using the initial model policy;
(c) constructing preference pairs at each node in each of the trajectories;
(d) adding the preference pairs to the replay buffer;
(e) updating the initial model policy creating an updated initial model policy based on the replay buffer, an executed plan of steps associated with the instruction executed in the updated initial model policy, a history of the executed plan and context of the executed plan, and the initial model policy;
(f) repeating steps (b)-(e) for a number of iterations to generate the optimized model policy based on the updated initial model policy;
(g) filtering the replay buffer based on a reward model or a verifier;
(h) iteratively refining the updated initial model policy based on a guided search strategy;
(i) producing a feedback score for each action based on a ranking of how much the action helped complete the instruction;
(j) computing a score for a node based on the feedback score and a reward;
and
(k) aggregating data from the trajectory in the replay buffer, wherein the replay buffer is off-policy,
wherein
updating the initial model policy is based on
ℒ
DPO
(
π
θ
;
𝒟
)
=
-
𝔼
(
T
ω
,
T
l
)
~
𝒟
[
og
⁢
σ
⁢
(
(
∑
t
=
0
i
⊤
⁢
ω
-
1
β
⁢
log
⁢
π
θ
(
a
t
ω
⁢
❘
"\[LeftBracketingBar]"
h
t
ω
)
π
ref
(
a
t
ω
⁢
❘
"\[LeftBracketingBar]"
h
t
ω
)
)
-
(
∑
t
=
0
i
⊤
↓
-
1
β
⁢
log
⁢
π
θ
(
a
t
l
⁢
❘
"\[LeftBracketingBar]"
h
t
l
)
π
ref
(
a
t
l
⁢
❘
"\[LeftBracketingBar]"
h
t
l
)
)
)
]
where D is the replay buffer of the trajectories, E is an expectation average operator, σ is a sigmoid function, β is a temperature parameter, a, is a plan of steps associated with the instruction executed in the updated initial model policy, ht is the history of the executed plan and the context of the executed plan, πθ is the updated initial model policy, πref is the initial model policy, t is an iteration number, the one or more agents comprise entities configured to perform actions in an environment,
the optimized model policy comprises one or more of a language model, a vision model, an audio model, a video model, an image model, a physical data model, or a haptic model,
the optimized model policy is based on an off-policy variant of a Direct Preference Optimization algorithm, and
the guided search strategy includes:
selecting the nodes in each of the trajectories;
generating a pre-selected number of completions for each of the nodes based on the updated initial model policy;
rolling out each of the trajectories based on the updated initial model policy until a terminal state is reached;
receiving the reward when the terminal state is reached;
and
backpropagating the reward multiplied by a discount factor based on
Q
⁡
(
s
t
,
a
)
←
Q
⁡
(
s
t
,
a
)
⁢
N
⁡
(
s
t
,
a
)
+
γ
T
-
t
⁢
R
N
⁡
(
s
t
,
a
)
+
1
N
⁡
(
s
t
,
a
)
←
N
⁡
(
s
t
,
a
)
+
1
,
where γ is the discount factor, s is a state, R is one of the trajectories, Q is the reward for passing through the state, a is an action, N is the number of times a state-action pair was visited during a search.

197 more elements

Dependencies are inferred from the claim text.

Combining search, self-critique and reinforcement learning for autonomous web agents

Life of this patent

Combining search, self-critique and reinforcement learning for autonomous web agents

Life of this patent

Abstract

Claims20