# US 2021/0334696 A1 · Training reinforcement machine learning systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0334696 A1 - Document ID: US20210334696A1 - Status: Published application - Kind: A1 - Application number: 16/859,874 - Filed: 2020-04-27 - Published: 2021-10-28 - Earliest claimed: 2020-04-27 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Eric Philip TRAUT; Marcos de Moura CAMPOS; Xuan ZHAO; Ross Ian STORY; Victor SHNAYDER - Web page: https://patentgrep.com/patent/US20210334696A1 ## CPC - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 20/00: Machine learning - G06F 11/3466: {Performance evaluation by tracing or monitoring} - G06F 11/3698: Environments for analysis, debugging or testing of software - G06N 3/08: Learning methods - G06N 3/092: Reinforcement learning - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 10/70: Quantum error correction, detection or prevention, e.g. surface codes or magic state distillation - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method of training a reinforcement machine learning computer system. The method comprises providing a machine-learning computer programming language including a pre-defined plurality of reinforcement machine learning criterion statements, and receiving a training specification authored in the machine-learning computer programming language. The training specification defines a plurality of training sub-goals with a corresponding plurality of the reinforcement machine learning criterion statements supported by the machine-learning computer programming language. The method further comprises computer translating the plurality of training sub-goals from the training specification into a shaped reward function configured to score a reinforcement machine learning model configuration with regard to the plurality of training sub-goals. The method further comprises running a training experiment with the reinforcement machine learning model configuration, scoring the reinforcement machine learning model in the training experiment with the shaped reward function, and adjusting the reinforcement machine learning model configuration based on the shaped reward function. ## Claims ### Claim 1 A method of training a reinforcement machine learning computer system, comprising: providing a machine-learning computer programming language including a pre-defined plurality of reinforcement machine learning criterion statements; receiving a training specification authored in the machine-learning computer programming language, wherein the training specification defines a plurality of training sub-goals with a corresponding plurality of the reinforcement machine learning criterion statements supported by the machine-learning computer programming language; computer translating the plurality of training sub-goals from the training specification into a shaped reward function configured to score a reinforcement machine learning model configuration with regard to the plurality of training sub-goals; running a training experiment using the reinforcement machine learning model configuration; scoring the reinforcement machine learning model in the training experiment with the shaped reward function; and adjusting the reinforcement machine learning model configuration based on an output from the shaped reward function. ### Claim 2 (depends on 1) The method of claim 1, wherein a training sub-goal of the plurality of training sub-goals includes a human-readable label, and wherein the training experiment is configured to log information pertaining to the human-readable label into an experiment log. ### Claim 3 (depends on 2) The method of claim 2, further comprising computer indexing the experiment log based on the human-readable label. ### Claim 4 (depends on 2) The method of claim 2, wherein the information pertaining to the human-readable label includes debugging information. ### Claim 5 (depends on 2) The method of claim 2, wherein the information pertaining to the human-readable label includes a performance metric. ### Claim 6 (depends on 1) The method of claim 1, wherein the shaped reward function is configured to increase a score for the reinforcement machine learning model configuration responsive to completing one of the training sub-goals. ### Claim 7 (depends on 1) The method of claim 1, wherein the shaped reward function is configured to decrease a score for the reinforcement machine learning model configuration responsive to failing one of the training sub-goals. ### Claim 8 (depends on 1) The method of claim 1, wherein the training experiment includes a default end, the method further comprising computer-assessing a termination condition based on the plurality of training sub-goals, and terminating the training experiment prior to the default end responsive to detecting the termination condition. ### Claim 9 (depends on 8) The method of claim 8, wherein computer-assessing the termination condition includes maintaining, for each training sub-goal of the plurality of training sub-goals, a success-flow graph representing a termination decision with regard to that sub-goal. ### Claim 10 (depends on 8) The method of claim 8, wherein the termination condition includes failing any sub-goal of the plurality of training sub-goals. ### Claim 11 (depends on 1) The method of claim 1, wherein the pre-defined plurality of reinforcement machine learning criterion statements includes a reach statement indicating a target aspect of the training experiment to be reached in the training experiment. ### Claim 12 (depends on 1) The method of claim 1, wherein the pre-defined plurality of reinforcement machine learning criterion statements includes an avoid statement indicating an aspect of the training experiment to be avoided in the training experiment. ### Claim 13 (depends on 1) The method of claim 1, wherein the pre-defined plurality of reinforcement machine learning criterion statements includes a drive statement indicating a maximum duration in which an aspect of the training experiment is allowed to deviate from a set boundary. ### Claim 14 (depends on 1) The method of claim 1, wherein the pre-defined plurality of reinforcement machine learning criterion statements includes an optimization statement indicating an aspect to be optimized in the training experiment. ### Claim 15 (depends on 1) The method of claim 1, further comprising assessing a performance metric for training based on the plurality of training sub-goals. ### Claim 16 (depends on 15) The method of claim 15, wherein the performance metric for training includes a frequency of successfully completing a sub-goal during the training experiment. ### Claim 17 (depends on 15) The method of claim 15, wherein the performance metric for training includes a criterion-specific performance metric for one of the pre-defined plurality of reinforcement machine learning criterion statements. ### Claim 18 (depends on 1) The method of claim 1, wherein translating the plurality of training sub-goals into the shaped reward function includes computer composing a plurality of sub-goal specific reward functions. ### Claim 19 A reinforcement machine learning computer system, comprising: an experiment instantiation machine configured to: receive a training specification authored in a machine-learning computer programming language including a pre-defined plurality of reinforcement machine learning criterion statements, wherein the training specification defines a plurality of training sub-goals with a corresponding plurality of the reinforcement machine learning criterion statements supported by the machine-learning computer programming language; and automatically translate the plurality of training sub-goals from the training specification into a shaped reward function configured to automatically score a reinforcement machine learning model configuration with regard to the plurality of training sub-goals; and an experiment execution machine configured to: run a training experiment of the reinforcement machine learning model configuration; score the reinforcement machine learning model in the training experiment with the shaped reward function; and adjust the reinforcement machine learning model configuration based on an output from the shaped reward function. ### Claim 20 A reinforcement machine learning computer system, comprising: a logic machine; and a storage machine holding instructions executable by the logic machine to: receive a training specification authored in a machine-learning computer programming language, wherein the training specification defines a plurality of training sub-goals with a corresponding plurality of reinforcement machine learning criterion statements supported by the machine-learning computer programming language; automatically translate the plurality of training sub-goals from the training specification into a shaped reward function configured to automatically score a reinforcement machine learning model configuration; run a training experiment with the reinforcement machine learning model configuration; and score the reinforcement machine learning model in the training experiment with the shaped reward function.