# US 12,117,917 B2 · Fair simultaneous comparison of parallel machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,117,917 B2 - Document ID: US12117917B2 - Status: Granted - Kind: B2 - Application number: 17/244,707 - Filed: 2021-04-29 - Published: 2022-11-24 - Granted: 2024-10-15 - Earliest claimed: 2021-04-29 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Robert Engel; Aly Megahed; Eric Kevin Butler; Nitin Ramchandani; Yuya Jeremy Ong - Web page: https://patentgrep.com/patent/US12117917B2 ## CPC - G06F 11/3409: {for performance assessment} - G06F 9/4843: {by program, e.g. task dispatcher, supervisor, operating system} - G06F 9/5005: {to service a request} - G06F 11/302: {where the computing system component is a software system} - G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)} - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/00: Machine learning - G06F 2201/865: Monitoring of software - G06F 2209/5011: Pool - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method of using a computing device to compare performance of multiple algorithms. The method includes receiving, by a computing device, multiple algorithms to assess. The computing device further receives a total amount of resources to allocate to the multiple algorithms. The computing device additionally assigns a fair share of the total amount of resources to each of the multiple algorithms. The computing device still further executes each of the multiple algorithms using the assigned fair share of the total amount of resources. The computing device additionally compares the performance of each of the multiple based on at least one of multiple hardware relative utility metrics describing a hardware relative utility of any given resource allocation for each of the multiple algorithms. ## Claims ### Claim 1 A method executed by a computing device comprising: receiving a plurality of machine learning (ML) algorithms; receiving a total amount of hardware resources; and executing each of the plurality of ML algorithms using a respective percentage of the total amount of hardware resources, wherein: input for each of the plurality of ML algorithms comprises a resource preference profile that includes a plurality of hardware relative utility metrics, the plurality of hardware relative utility metrics is based on a resource allocation index, a resource distribution index, and a resource variance index, bias stemming from different hardware requirements or preferences for each of the plurality of ML algorithms is eliminated or mitigated, the respective performance of each of the plurality of ML algorithms associated with the execution is based on at least one of the plurality of hardware relative utility metrics describing a hardware relative utility of any given resource allocation for each of the plurality of ML algorithms, and a particular ML solution is based on the plurality of ML algorithms. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of ML algorithms comprises, simulation algorithms or search algorithms. ### Claim 3 (depends on 2) The method of claim 2, wherein the performance of each of the plurality of ML algorithms is measured in terms of time of execution, accuracy, or a combination thereof. ### Claim 4 (depends on 2) The method of claim 2, wherein the input for each of the plurality of ML algorithms further comprises a set of resource types and a resource bundle that includes required hardware resources for the plurality of ML algorithms. ### Claim 5 (depends on 4) The method of claim 4, wherein an optimum overall hardware resource allocation for the plurality of ML algorithms is based on a pareto-optimal, equitable, or relaxed-equitable distribution of relative hardware utilities across the plurality of ML algorithms according to the plurality of hardware relative utility metrics. ### Claim 6 (depends on 1) The method of claim 1, wherein an optimal allocation of hardware resources to each of the plurality of ML algorithms is based on multiplicities of hardware resource bundles. ### Claim 7 A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: receive a plurality of machine learning (ML) algorithms; receive a total amount of hardware resources; and execute each of the plurality of ML algorithms using a respective percentage of the total amount of hardware resources, wherein: input for each of the plurality of ML algorithms comprises a resource preference profile that includes a plurality of hardware relative utility metrics, the plurality of hardware relative utility metrics is based on a resource allocation index, a resource distribution index, and a resource variance index, bias stemming from different hardware requirements or preferences for each of the plurality of ML algorithms is eliminated or mitigated, the respective performance of each of the plurality of ML algorithms associated with the execution is based on at least one of the plurality of hardware relative utility metrics describing a hardware relative utility of any given resource allocation for each of the plurality of ML algorithms, and a particular ML solution is based on the plurality of ML algorithms. ### Claim 8 (depends on 7) The computer program product of claim 7, wherein the plurality of ML algorithms comprises simulation algorithms or search algorithms. ### Claim 9 (depends on 8) The computer program product of claim 8, wherein the performance of each of the plurality of ML algorithms is measured in terms of time of execution, accuracy, or a combination thereof. ### Claim 10 (depends on 8) The computer program product of claim 8, wherein the input for each of the plurality of ML algorithms further comprises a set of resource types and a resource bundle that includes required hardware resources for the plurality of ML algorithms. ### Claim 11 (depends on 10) The computer program product of claim 10, wherein an optimum overall hardware resource allocation for the plurality of ML algorithms is based on a pareto-optimal, equitable, or relaxed-equitable distribution of relative hardware utilities across the plurality of ML algorithms according to the plurality of hardware relative utility metrics. ### Claim 12 (depends on 7) The computer program product of claim 7, wherein an optimal allocation of hardware resources to each of the plurality of ML algorithms is based on multiplicities of hardware resource bundles. ### Claim 13 An apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to: receive a plurality of machine learning (ML) algorithms; receive a total amount of hardware resources; and execute each of the plurality of ML algorithms using a respective percentage of the total amount of hardware resources, wherein: input for each of the plurality of ML algorithms comprises a resource preference profile that includes a plurality of hardware relative utility metrics, the plurality of hardware relative utility metrics is based on a resource allocation index, a resource distribution index, and a resource variance index, bias stemming from different hardware requirements or preferences for each of the plurality of ML algorithms is eliminated or mitigated, the respective performance of each of the plurality of ML algorithms associated with the execution is based on at least one of the plurality of hardware relative utility metrics describing a hardware relative utility of any given resource allocation for each of the plurality of ML algorithms, and a particular ML solution is based on the plurality of ML algorithms. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the plurality of ML algorithms comprises simulation algorithms or search algorithms. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the performance of each of the plurality of ML algorithms is measured in terms of time of execution, accuracy, or a combination thereof. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein the input for each of the plurality of ML algorithms further comprises a set of resource types and a resource bundle that includes required hardware resources for the plurality of ML algorithms. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein an optimum overall hardware resource allocation for the plurality of ML algorithms is based on a pareto-optimal, equitable, or relaxed-equitable distribution of relative hardware utilities across the plurality of ML algorithms according to the plurality of hardware relative utility metrics. ### Claim 18 (depends on 13) The apparatus of claim 13, wherein an optimal allocation of hardware resources to each of the plurality of ML algorithms is based on multiplicities of hardware resource bundles.