# US 2021/0241167 A1 · Computing system for training, deploying, executing, and updating machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241167 A1 - Document ID: US20210241167A1 - Status: Published application - Kind: A1 - Application number: 16/780,853 - Filed: 2020-02-03 - Published: 2021-08-05 - Earliest claimed: 2020-02-03 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Paul Luo LI; Ho Jeannie CHUNG; Xiaoyu CHAI; Irina Ioana NICULESCU; Minsuk KANG; Brandon H. PADDOCK; Jilong LIAO; Neeraja ABBURU; James Henry DOOLEY, IV; Frederick Douglass CAMPBELL - Web page: https://patentgrep.com/patent/US20210241167A1 ## CPC - G06N 20/00: Machine learning - G06F 8/65: Updates (security arrangements therefor G06F21/57) - G06F 18/10: Pre-processing; Data cleansing - G06F 8/60: Software deployment - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A server computing device generates training data based upon an identifier for a device, a timestamp, and a label received from a developer computing device. The server computing device trains a computer-implemented machine learning (ML) model based upon the training data. The server computing device also generates client configuration data for the ML model that specifies transformations that are to be applied to values in order to generate input values for the ML model. The server computing device deploys ML assets to client computing devices, the ML assets comprising the ML model and the client configuration data. The client computing devices execute the ML model using input values derived via transformations of (local) values produced by the client computing devices and transmit telemetry data to the server computing device. The server computing device updates the ML assets based upon the telemetry data. ## Claims ### Claim 1 A server computing device, comprising: a processor; and memory storing instructions that, when executed by the processor, cause the processor to perform acts comprising: receiving an identity of a file that includes an identifier for a device, a timestamp, and a label from a developer computing device, wherein the label identifies a prediction that is to be made, wherein a target application is to utilize the prediction; generating training data based upon the identifier for the device, the timestamp, and the label, wherein generating the training data comprises generating values for a predefined set of features; training a computer-implemented machine learning (ML) model based upon the training data, wherein the ML model is configured to take input values for client-generated features as input, wherein the ML model is configured to output the prediction based upon the input values; generating client configuration data that comprises transformations that are to be applied to values produced by client computing devices in order to generate the input values; deploying ML assets to the client computing devices, the ML assets comprising the ML model and the client configuration data, wherein the client computing devices transform the values produced by the client computing devices to the input values based upon the client configuration data, wherein the client computing devices provide the input values as input to the NIL model, wherein the ML model outputs the prediction based upon the input values; and subsequent to deploying the ML assets to the client computing devices, updating the ML assets based upon telemetry data received from the client computing devices. ### Claim 2 (depends on 1) The server computing device of claim 1, the acts further comprising: deploying the updated ML assets to a portion of the client computing devices, wherein the server computing device assesses performance of the updated ML assets based upon a comparison between the telemetry data and second telemetry data produced by the portion of the client computing devices when the portion of the client computing devices execute an updated ML model in the ML assets. ### Claim 3 (depends on 1) The server computing device of claim 1, wherein the client configuration data further comprises identifiers for features that are to be retrieved by the client computing device, wherein the client computing device retrieves the values for the features based upon the client configuration data. ### Claim 4 (depends on 1) The server computing device of claim 1, wherein a software developer is developing the target application on the developer computing device, wherein the developer computing device receives the identity of the file, the timestamp, and the label from the software developer. ### Claim 5 (depends on 1) The server computing device of claim 1, the acts further comprising: assigning a scenario identifier to the ML assets. ### Claim 6 (depends on 5) The server computing device of claim 5, wherein the target application executes on a client computing device in the client computing devices, wherein the target application performs an application programming interface (API) call using the scenario identifier, wherein the API call causes the client computing device to execute the ML model. ### Claim 7 (depends on 1) The server computing device of claim 1, wherein a client computing device executes the ML model based upon input values produced locally at the client computing device, wherein the target application performs a first action when the prediction output by the ML model is above a threshold value, wherein the target application performs a second action when the prediction output by the ML model is below the threshold value. ### Claim 8 (depends on 1) The server computing device of claim 1, wherein updating the ML assets based upon the telemetry data comprises: removing a feature from the predefined set of features; and retraining the ML model based the predefined set of features with the feature removed therefrom. ### Claim 9 (depends on 1) The server computing device of claim 1, wherein the file includes a record of user input to the device received by the operating system of the device or a record of user input received by the target application. ### Claim 10 (depends on 1) The server computing device of claim 1, wherein the predefined set of features include patterns of input received by each of the client computing devices from users of each of the client computing devices. ### Claim 11 A method executed by a processor of a server computing device, comprising: receiving an identity of a file, a timestamp, and a label from a developer computing device, wherein the label identifies a prediction that is to be made, wherein a target application is to utilize the prediction; generating training data based upon the identity of the file, the timestamp, and the label, wherein generating the training data comprises generating values for a predefined set of features; training a computer-implemented machine learning (ML) model based upon the training data, wherein the ML model is configured to take input values for client-generated features as input, wherein the ML model is configured to output the prediction based upon the input values; generating client configuration data that comprises transformations that are to be applied to values produced by client computing devices in order to generate the input values; deploying ML assets to the client computing devices, the ML assets comprising the ML model and the client configuration data, wherein the client computing devices transform the values produced by the client computing devices to the input values based upon the client configuration data, wherein the client computing devices provide the input values as input to the ML model, wherein the ML model outputs the prediction based upon the input values; and subsequent to deploying the ML assets to the client computing devices, updating the ML assets based upon telemetry data received from the client computing devices. ### Claim 12 (depends on 11) The method of claim 11, wherein updating the ML assets based upon the telemetry data comprises: adding an additional feature to the predefined set of features; and retraining the ML model based upon the predefined set of features that includes the additional feature. ### Claim 13 (depends on 11) The method of claim 11, wherein the predefined features include patterns of input received by each of the client computing devices from users of each of the client computing devices. ### Claim 14 (depends on 11) The method of claim 11, wherein the telemetry data comprises user feedback from users of the client computing devices, wherein updating the ML assets is further based upon the user feedback. ### Claim 15 (depends on 11) The method of claim 11, wherein the server computing device is a cloud-based computing platform. ### Claim 16 (depends on 11) The method of claim 11, wherein the ML assets are deployed to the client computing devices as part of an operating system update to the client computing devices. ### Claim 17 (depends on 11) The method of claim 11, further comprising: deploying the updated ML assets to a portion of the client computing devices, wherein the server computing device assesses performance of the updated ML assets based upon a comparison between the telemetry data and second telemetry data produced by the portion of the client computing devices when the portion of the client computing devices execute an updated ML model in the ML assets; and when the comparison indicates that the updated ML model generated correct prediction at a greater frequency than the ML model, deploying the updated ML assets to client computing devices in the client computing devices that did not receive the updated ML assets. ### Claim 18 A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising: receiving an identity of a file that includes an identifier for a device, a timestamp, and a label from a developer computing device, wherein the label identifies a prediction that is to be made, where a target application is to utilize the prediction; generating training data based upon the identifier for the device, the timestamp, and the label, wherein generating the training data comprises generating values for a predefined set of features; training a computer-implemented machine learning (ML) model based upon the training data, wherein the ML model is configured to take input values for client-generated features as input, wherein the ML model is configured to output the prediction based upon the input values; generating client configuration data that comprises transformations that are to be applied to values produced by client computing devices in order to generate the input values; deploying ML assets to the client computing devices, the ML assets comprising the ML model and the client configuration data, wherein the client computing devices transform the values produced by the client computing devices to the input values based upon the client configuration data, wherein the client computing devices provide the input values as input to the NIL model, wherein the ML model outputs the prediction based upon the input values; and subsequent to deploying the ML assets to the client computing devices, updating the ML assets based upon telemetry data received from the client computing devices. ### Claim 19 (depends on 18) The computer-readable storage medium of claim 18, wherein the target application performs an action based upon the prediction output by the ML model when the ML model is executed by a client computing device in the client computing devices. ### Claim 20 (depends on 18) The computer-readable storage medium of claim 18, wherein the ML assets are deployed to the client computing devices as part of an operating system update to the client computing devices.