# US 2025/0165796 A1 · Systems and methods for executing confidence-aware reinforcement learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0165796 A1 - Document ID: US20250165796A1 - Status: Published application - Kind: A1 - Application number: 18/516,125 - Filed: 2023-11-21 - Published: 2025-05-22 - Earliest claimed: 2023-11-21 - Applicants: Huawei Technologies Canada Co., Ltd. - Inventors: Pascal POUPART; Guiliang LIU; Mohammed ELMAHGIUBI - Web page: https://patentgrep.com/patent/US20250165796A1 ## CPC - G06N 3/092: Reinforcement learning - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods and systems for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment are disclosed. The method includes accessing a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, the expert entities complying with an expert constraint that is unknown. The method also includes generating a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model, determining a target policy among a plurality of policies, the target policy complying with the main constraint and executing the target policy by the AI model. ## Claims ### Claim 1 A computer-implemented method for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, the method comprising: accessing a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, a given one of the expert trajectories including information about a given state of the environment and a corresponding action that is to be executed in response to the given state, the expert entities complying with an expert constraint that is unknown; generating a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model; determining a target policy among a plurality of policies, the target policy complying with the main constraint; and executing the target policy by the AI model. ### Claim 2 (depends on 1) The method of claim 1, further comprising: accessing a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories, an execution of the policy aiming at maximizing a reward; determining a policy complying with the main constraint; executing the policy by iteratively: executing the actions of the policy, receiving indication of rewards from and states of the environment, and adjusting the policy based on outcomes of the actions and received rewards. ### Claim 3 (depends on 2) The method of claim 2, further comprising, prior to executing the policy: determining a policy-value of the target policy; in response to the policy-value being below a pre-determined value threshold, flagging the set of expert trajectories as insufficient. ### Claim 4 (depends on 3) The method of claim 3, further comprising augmenting the set of expert trajectories with additional expert trajectories until the policy-value exceeds the pre-determined value threshold. ### Claim 5 (depends on 3) The method of claim 3, wherein determining the policy-value of the policy comprises determining an expected cumulative reward based on rewards associated with the action-state pairs of the policy. ### Claim 6 (depends on 1) The method of claim 1, wherein generating a main constraint for the set of expert trajectories comprises: determining a constraint distribution based on the set of expert trajectories; selecting a constraint from the constraint distribution based on the pre-determined confidence level as the main constraint. ### Claim 7 (depends on 6) The method of claim 6, wherein selecting the constraint from the constraint distribution comprises selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level. ### Claim 8 (depends on 7) The method of claim 7, wherein the main constraint is: quantileP(c)(1−λ) where P(c) is the constraint distribution and λ is the pre-determined confidence level. ### Claim 9 (depends on 6) The method of claim 6, wherein determining a constraint distribution comprises: employing a neural network encoding the set of expert trajectories to determine, for each of the expert trajectory, a set of contribution factors; and adjusting a template distribution according to the set of contribution factors to form the constraint distribution. ### Claim 10 (depends on 9) The method of claim 9, wherein each expert trajectory is encoded with a corresponding encoder having corresponding weights in the neural network. ### Claim 11 A system for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, the system comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller, cause the system to: access a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, a given one of the expert trajectories including information about a given state of the environment and a corresponding action that is to be executed in response to the given state, the expert entities complying with an expert constraint that is unknown; generate a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model; determine a target policy among a plurality of policies, the target policy complying with the main constraint; and execute the target policy by the AI model. ### Claim 12 (depends on 11) The system of claim 11, wherein the system is further configured to: access a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories, an execution of the policy aiming at maximizing a reward; determine a policy complying with the main constraint; execute the policy by iteratively: executing the actions of the policy, receiving indication of rewards from and states of the environment, and adjusting the policy based on outcomes of the actions and received rewards. ### Claim 13 (depends on 12) The system of claim 12, wherein the system is further configured to, prior to executing the policy: determine a policy-value of the target policy; in response to the policy-value being below a pre-determined value threshold, flag the set of expert trajectories as insufficient. ### Claim 14 (depends on 13) The system of claim 13, wherein the system is further configured to augment the set of expert trajectories with additional expert trajectories until the policy-value exceeds the pre-determined value threshold. ### Claim 15 (depends on 13) The system of claim 13, wherein the system is further configured to, upon determining the policy-value of the policy, determine an expected cumulative reward based on rewards associated with the action-state pairs of the policy. ### Claim 16 (depends on 11) The system of claim 11, wherein the system is further configured to, upon generating a main constraint for the set of expert trajectories: determine a constraint distribution based on the set of expert trajectories; select a constraint from the constraint distribution based on the pre-determined confidence level as the main constraint. ### Claim 17 (depends on 16) The system of claim 16, wherein the system is further configured to select the constraint from the constraint distribution by selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level. ### Claim 18 (depends on 17) The system of claim 17, wherein the main constraint is: quantileP(c)(1−λ) where P(c) is the constraint distribution and λ is the pre-determined confidence level. ### Claim 19 (depends on 16) The system of claim 16, wherein the system is further configured to, upon determining a constraint distribution: employ a neural network encoding the set of expert trajectories to determine, for each of the expert trajectory, a set of contribution factors; and adjust a template distribution according to the set of contribution factors to form the constraint distribution. ### Claim 20 (depends on 19) The system of claim 19, wherein each expert trajectory is encoded with a corresponding encoder having corresponding weights in the neural network.