# US 2020/0401491 A1 · Framework for testing machine learning workflows > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401491 A1 - Document ID: US20200401491A1 - Status: Published application - Kind: A1 - Application number: 16/447,739 - Filed: 2019-06-20 - Published: 2020-12-24 - Earliest claimed: 2019-06-20 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Ali Sadiq Mohamed; Manish Swaminathan; Shunlin Liang; Prateek Sachdev; Vivek Desai; Adam R. Peck; Sunny Sanjiv Ketkar - Web page: https://patentgrep.com/patent/US20200401491A1 ## CPC - G06F 11/2273: {Test methods} - G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment} - G06N 20/00: Machine learning - G06N 3/08: Learning methods - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The disclosed embodiments provide a system for testing machine learning workflows. During operation, the system obtains a configuration for a staging test of a machine learning model, wherein the configuration includes a model name for the machine learning model, a duration of the staging test, and a use case associated with the machine learning model. Next, the system selects a staging test host for the staging test. The system then deploys the staging test on the staging test host in a staging environment, wherein the deployed staging test executes the machine learning model based on live traffic received from a production environment. After the staging test has completed, the system outputs a set of metrics representing a system impact of the machine learning model on the staging test host. ## Claims ### Claim 1 A method, comprising: obtaining a configuration for a staging test of a machine learning model, wherein the configuration comprises a model name for the machine learning model, a duration of the staging test, and a use case associated with the machine learning model; selecting, by one or more computer systems, a staging test host for the staging test; deploying, by the one or more computer systems, the staging test on the staging test host in a staging environment, wherein the deployed staging test executes the machine learning model based on live traffic received from a production environment; and after the staging test has completed, outputting a set of metrics representing a system impact of the machine learning model on the staging test host. ### Claim 2 (depends on 1) The method of claim 1, further comprising: upon detecting a failure in the staging test host during execution of the staging test on the staging test host, retrying the staging test on another staging test host in the staging environment. ### Claim 3 (depends on 2) The method of claim 2, further comprising: tracking a number of retries of the staging test in the staging environment; and discontinuing the staging test after the number of retries reaches a limit. ### Claim 4 (depends on 1) The method of claim 1, wherein selecting the staging test host for the staging test comprises: tracking states of a set of staging test hosts in the staging environment; and selecting, for the staging test, the staging test host with a state that indicates availability to perform testing. ### Claim 5 (depends on 4) The method of claim 4, wherein the states of the set of staging hosts further indicate at least one of: current execution of a test; and unavailability to perform testing. ### Claim 6 (depends on 1) The method of claim 1, wherein selecting the staging test host for the staging test comprises: storing a set of staging tests comprising the staging test in a queue; and selecting staging test hosts for the set of staging tests according to an ordering of the set of staging tests in the queue. ### Claim 7 (depends on 1) The method of claim 1, wherein executing the machine learning model based on live traffic received from the production environment comprises: upon detecting a shift in the live traffic from a first location comprising the staging test host to a second location that is external to the staging test host, moving the staging test to another staging test host in the second location. ### Claim 8 (depends on 1) The method of claim 1, wherein executing the machine learning model based on live traffic received from the production environment comprises: generating features based on user attributes, user activity, and content items in the live traffic; and applying the machine learning model to the features to generate scores representing predicted likelihoods of interacting with the content items. ### Claim 9 (depends on 8) The method of claim 8, wherein the use case comprises at least one of: an access point for a content feed comprising a ranking of the content items by the scores. ### Claim 10 (depends on 1) The method of claim 1, wherein executing the machine learning model based on live traffic received from the production environment comprises: transmitting the live traffic from a first host in a first location of the production environment to the staging test host; and after the live traffic has shifted from the first location to a second location of the production environment, transmitting the live traffic from a second host in the second location to the staging test host. ### Claim 11 (depends on 1) The method of claim 1, wherein the configuration further comprises a proportion of the live traffic to direct to the machine learning model in the staging environment. ### Claim 12 (depends on 1) The method of claim 1, wherein the configuration further comprises an A/B test that compares the system impact of the machine learning model with an additional system impact of an additional machine learning model. ### Claim 13 (depends on 1) The method of claim 1, wherein the set of metrics comprises at least one of: a latency; a queries per second (QPS); a garbage collection duration; a memory usage; a processor utilization; a garbage collection footprint; an error rate; a timeout count; and a percentile. ### Claim 14 A system, comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the system to: obtain a configuration for a staging test of a machine learning model, wherein the configuration comprises a model name for the machine learning model, a duration of the staging test, and a use case associated with the machine learning model; select a staging test host for the staging test; deploy the staging test on the staging test host in a staging environment, wherein the deployed staging test executes the machine learning model based on live traffic received from a production environment; and after the staging test has completed, output a set of metrics representing a system impact of the machine learning model on the staging test host. ### Claim 15 (depends on 14) The system of claim 14, wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to: track a number of retries of the staging test in the staging environment; upon detecting a failure in the staging test host during execution of the staging test on the staging test host, retry the staging test on another staging test host in the staging environment; and discontinue the staging test after the number of retries reaches a limit. ### Claim 16 (depends on 14) The system of claim 14, wherein selecting the staging test host for the staging test comprises: tracking states of a set of staging test hosts in the staging environment, wherein the states comprise at least one of availability to perform testing, current execution of a test, and unavailability to perform testing; and selecting, for the staging test, the staging test host with a state that indicates the availability to perform testing. ### Claim 17 (depends on 14) The system of claim 14, wherein selecting the staging test host for the staging test comprises: storing a set of staging tests comprising the staging test in a queue; and selecting staging test hosts for the set of staging tests according to an ordering of the set of staging tests in the queue. ### Claim 18 (depends on 14) The system of claim 14, wherein executing the machine learning model based on live traffic received from the production environment comprises: upon detecting a shift in the live traffic from a first location comprising the staging test host to a second location that is external to the staging test host, moving the staging test to another staging test host in the second location. ### Claim 19 (depends on 14) The system of claim 14, wherein the configuration further comprises at least one of: a proportion of the live traffic to direct to the machine learning model in the staging environment; and an A/B test that compares the system impact of the machine learning model with an additional system impact of an additional machine learning model. ### Claim 20 A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising: obtaining a configuration for a staging test of a machine learning model, wherein the configuration comprises a model name for the machine learning model, a duration of the staging test, and a use case associated with the machine learning model; selecting a staging test host for the staging test; deploying the staging test on the staging test host in a staging environment, wherein the deployed staging test executes the machine learning model based on live traffic received from a production environment; and after the staging test has completed, outputting a set of metrics representing a system impact of the machine learning model on the staging test host.