# US 2019/0378052 A1 · Automated versioning and evaluation of machine learning workflows > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378052 A1 - Document ID: US20190378052A1 - Status: Published application - Kind: A1 - Application number: 16/184,050 - Filed: 2018-11-08 - Published: 2019-12-12 - Earliest claimed: 2018-06-06 - Assignees: EmbodyVR, Inc. - Applicants: EmbodyVR, Inc. - Inventors: Shruti Badhwar; Faiyadh Shahid; Nikhil Ratnesh Javeri; Kapil Jain - Web page: https://patentgrep.com/patent/US20190378052A1 ## CPC - G06N 20/00: Machine learning - G06F 9/44505: {Configuring for program initiating, e.g. using registry, configuration files} - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 3/08: Learning methods - G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An application such as a machine learning workflow manager executes a machine learning workflow utilizing varying sets of parameters and tracks the utilized parameters and performance metrics for each execution. The manager generates a unique experiment identifier (ID) for each set of parameters used for executing a workflow and stores the experiment ID and the set of parameters along with results of the execution, e.g., performance metrics, output data, program code, etc. In some implementations, the manager can algorithmically generate sets of parameters for experiments of the workflow. Once experimentation for the workflow is complete, the manager utilizes the stored performance metrics to identify an experiment which exhibited the best performance and can retrieve the set of parameters, output data, or generated program code for deployment of the workflow using the associated experiment ID. ## Claims ### Claim 1 A method comprising: determining a first configuration for a first machine learning workflow; generating a first identifier for the first configuration based, at least in part, on one or more parameters indicated in the first configuration; executing the first machine learning workflow in accordance with the first configuration; and recording results of the execution of the first machine learning workflow in accordance with the first configuration along with the first identifier. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the first identifier for the first configuration based, at least in part, on the one or more parameters indicated in the first configuration comprises: normalizing the one or more parameters from the first configuration; and encoding the one or more parameters to generate an identifier which is unique to the one or more parameters of the first configuration. ### Claim 3 (depends on 1) The method of claim 1 further comprising: prior to executing the first machine learning workflow, comparing the first identifier to a plurality of identifiers corresponding to previously utilized configurations; and determining that the first identifier does not match any of the plurality of identifiers. ### Claim 4 (depends on 1) The method of claim 1 further comprising: generating a second identifier for a second configuration for the first machine learning workflow; comparing the second identifier to a plurality of identifiers corresponding to previously utilized configurations; and based on determining that the second identifier matches an identifier in the plurality of identifiers, preventing execution of the first machine learning workflow in accordance with the second configuration. ### Claim 5 (depends on 1) The method of claim 1 further comprising: executing the first machine learning workflow in accordance with a second configuration; and determining an optimal configuration for the first machine learning workflow based, at least in part, on comparing results of the execution of the first machine learning workflow in accordance with the second configuration to the results of the execution of the first machine learning workflow in accordance with the first configuration. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the first configuration for the first machine learning workflow comprises: determining a set of potential values for at least a first parameter and a second parameter for the first machine learning workflow; and selecting, from the set of potential values, a value for the first parameter and a value for the second parameter. ### Claim 7 (depends on 6) The method of claim 6 further comprising generating a plurality of configurations for the first machine learning workflow based, at least in part, on determining unique combinations of values for the first parameter and the second parameter from the set of potential values. ### Claim 8 (depends on 1) The method of claim 1, wherein executing the first machine learning workflow in accordance with the first configuration comprises: instantiating one or more machine learning models identified in the first configuration within a container environment; and inputting data identified in the first configuration into the one or more machine learning models. ### Claim 9 (depends on 1) The method of claim 1, wherein recording the results of the first machine learning workflow comprises recording at least one of performance metrics, program code, and output data. ### Claim 10 A non-transitory, computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations comprising: determining a set of configurations for one or more tasks of a machine learning workflow; for each configuration in the set of configurations, generating an identifier for the configuration based, at least in part, on parameters indicated in the configuration; executing the machine learning workflow in accordance with the configuration; and recording metrics related to the execution of the machine learning workflow in association with the identifier; and identifying an optimal configuration from the set of configurations for the machine learning workflow based, at least in part, on the metrics. ### Claim 11 (depends on 10) The computer-readable medium of claim 10, wherein the one or more tasks comprise tasks for performing audio spatialization, wherein at least a first configuration of the set of configurations identifies machine learning models to be used for generating a head response transfer function prediction for the audio spatialization. ### Claim 12 (depends on 11) The computer-readable medium of claim 11, wherein the tasks for performing the audio spatialization comprise an object detection and segmentation task, a feature extraction task, and a head response transfer function prediction task. ### Claim 13 An apparatus comprising: a database comprising identifiers associated with executions of a first machine learning workflow; a model repository comprising a plurality of machine learning models; a processor; and a computer-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to, determine a first configuration for the first machine learning workflow, wherein the first configuration specifies one or more models in the model repository; generate a first identifier for the first configuration based, at least in part, on one or more parameters indicated in the first configuration; execute the first machine learning workflow in accordance with the first configuration; and store results of the execution of the first machine learning workflow in accordance with the first configuration along with the first identifier in the database. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the instructions to generate the first identifier for the first configuration based, at least in part, on the one or more parameters indicated in the first configuration comprises instructions to: normalize the one or more parameters from the first configuration; and encode the one or more parameters to generate an identifier which is unique to the one or more parameters of the first configuration. ### Claim 15 (depends on 13) The apparatus of claim 13 further comprising instructions to: prior to execution the first machine learning workflow, compare the first identifier to a plurality of identifiers corresponding to previously utilized configurations; and determine that the first identifier does not match any of the plurality of identifiers. ### Claim 16 (depends on 13) The apparatus of claim 13 further comprising instructions to: generate a second identifier for a second configuration for the first machine learning workflow; compare the second identifier to a plurality of identifiers corresponding to previously utilized configurations; and based on a determination that the second identifier matches an identifier in the plurality of identifiers, prevent execution of the first machine learning workflow in accordance with the second configuration. ### Claim 17 (depends on 13) The apparatus of claim 13 further comprising instructions to: execute the first machine learning workflow in accordance with a second configuration; and determine an optimal configuration for the first machine learning workflow based, at least in part, on comparing results of the execution of the first machine learning workflow in accordance with the second configuration to the results of the execution of the first machine learning workflow in accordance with the first configuration. ### Claim 18 (depends on 13) The apparatus of claim 13, wherein the instructions to determine the first configuration for the first machine learning workflow comprises instructions to: determine a set of potential values for at least a first parameter and a second parameter for the first machine learning workflow; and select, from the set of potential values, a value for the first parameter and a value for the second parameter. ### Claim 19 (depends on 18) The apparatus of claim 18 further comprising instructions to generate a plurality of configurations for the first machine learning workflow based, at least in part, on determining unique combinations of values for the first parameter and the second parameter from the set of potential values. ### Claim 20 (depends on 13) The apparatus of claim 13, wherein the instructions to executing the first machine learning workflow in accordance with the first configuration comprises instructions to: instantiate one or more machine learning models identified in the first configuration within a container environment; and input data identified in the first configuration into the one or more machine learning models.