# US 11,907,810 B2 · Concurrent optimization of machine learning model performance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,810 B2 - Document ID: US11907810B2 - Status: Granted - Kind: B2 - Application number: 16/515,711 - Filed: 2019-07-18 - Published: 2021-01-21 - Granted: 2024-02-20 - Earliest claimed: 2019-07-18 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Serag Gadelrab; James Esliger; Meghal Varia; Kyle Ernewein; Alwyn Dos Remedios; George Lee - Web page: https://patentgrep.com/patent/US11907810B2 ## CPC - G06N 20/00: Machine learning - G06F 11/3409: {for performance assessment} - G06F 11/3447: {Performance evaluation by modeling} - G06F 11/3466: {Performance evaluation by tracing or monitoring} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 5/04: Inference or reasoning models - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Certain aspects of the present disclosure provide techniques for concurrently performing inferences using a machine learning model and optimizing parameters used in executing the machine learning model. An example method generally includes receiving a request to perform inferences on a data set using the machine learning model and performance metric targets for performance of the inferences. At least a first inference is performed on the data set using the machine learning model to meet a latency specified for generation of the first inference from receipt of the request. While performing the at least the first inference, operational parameters resulting in inference performance approaching the performance metric targets are identified based on the machine learning model and operational properties of the computing device. The identified operational parameters are applied to performance of subsequent inferences using the machine learning model. ## Claims ### Claim 1 A method for performing inferences using a machine learning model on a computing device, comprising: receiving a request to perform inferences on a data set using a machine learning model configured to perform the inferences and performance metric targets for performance of the inferences; performing a first inference on the data set using the machine learning model to meet a latency specified for generation of the first inference from receipt of the request, wherein the first inference is performed using baseline operational parameters configured for execution of inference operations using the machine learning model, wherein the baseline operational parameters comprise predefined parameters for the machine learning model; while performing the first inference using the machine learning model, simultaneously identifying operational parameters using an operational parameter optimizer model configured to identify the operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets based on the machine learning model and operational properties of a computing device; and applying the identified operational parameters to performance of subsequent inferences using the machine learning model. ### Claim 2 (depends on 1) The method of claim 1, wherein the operational parameters comprise parameters associated with hardware parameters or hardware conditions at the computing device. ### Claim 3 (depends on 1) The method of claim 1, wherein the baseline operational parameters are identified by: generating a representation of the machine learning model; and searching a repository for operational parameters associated with the representation of the machine learning model and the operational properties of the computing device. ### Claim 4 (depends on 1) The method of claim 1, wherein performing the at least the first inference on the data set comprises: estimating a total amount of time to identify the baseline operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets and generate the first inference on the data set using the baseline operational parameters; determining that the estimated total amount of time is within the latency specified for generation of the first inference from receipt of the request; and performing the first inference on the data set by applying the baseline operational parameters to execution of the machine learning model. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying operational parameters resulting in inference performance approaching the performance metric targets based on the machine learning model and operational properties of the computing device comprises: generating a plurality of sets of operational parameters, each set being associated with a different performance metric; and selecting one of the plurality of sets of operational parameters having a performance metric closest to the performance metric targets. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting a change in the operational properties of the computing device; updating the operational parameters based on the machine learning model and the changed operational properties of the computing device; and applying the updated operational parameters to performance of additional inferences using the machine learning model. ### Claim 7 (depends on 6) The method of claim 6, wherein detecting the change in the operational properties of the computing device comprises determining that a difference between previous operational properties of the computing device and current operational properties of the computing device exceeds a threshold amount of change. ### Claim 8 (depends on 1) The method of claim 1, wherein identifying operational parameters resulting in inference performance approaching the performance metric targets based on the machine learning model and operational properties of the computing device comprises inputting information about the machine learning model and the operational properties of the computing device into the operational parameter optimizer model trained using a training data set of performance data from previous inferences and performance metrics for the previous inferences. ### Claim 9 (depends on 8) The method of claim 8, further comprising: for each inference performed on the computing device, augmenting the training data set with performance data for each inference and operational parameters applied to the machine learning model for each inference. ### Claim 10 (depends on 9) The method of claim 9, further comprising: periodically retraining the operational parameter optimizer model using the augmented training data set. ### Claim 11 (depends on 8) The method of claim 8, further comprising: periodically updating the operational parameter optimizer model based on updates received from a remote system. ### Claim 12 (depends on 1) The method of claim 1, wherein identifying operational parameters resulting in inference performance approaching the performance metric targets based on the machine learning model and operational properties of the computing device comprises: transmitting, to the operational parameter optimizer model executing on a remote system, information about the machine learning model and the operational properties of the computing device; and receiving the operational parameters from the operational parameter optimizer model executing on the remote system. ### Claim 13 A system, comprising: one or more processors; and a memory having instructions stored thereon which, when executed by the one or more processors configured individually or in any combination, performs an operation for performing inferences using a machine learning model on a computing device, the operation comprising: receiving a request to perform inferences on a data set using a machine learning model configured to perform the inferences and performance metric targets for performance of the inferences; performing a first inference on the data set using the machine learning model to meet a latency specified for generation of the first inference from receipt of the request, wherein the first inference is performed using baseline operational parameters configured for execution of inference operations using the machine learning model, wherein the baseline operational parameters comprise predefined parameters for the machine learning model; while performing the first inference using the machine learning model, simultaneously identifying operational parameters using an operational parameter optimizer model configured to identify the operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets based on the machine learning model and operational properties of a computing device; and applying the identified operational parameters to performance of subsequent inferences using the machine learning model. ### Claim 14 (depends on 13) The system of claim 13, wherein performing the at least the first inference on the data set comprises: estimating a total amount of time to identify the baseline operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets and generate the first inference on the data set using the baseline operational parameters; determining that the estimated total amount of time is within the latency specified for generation of the first inference from receipt of the request; and performing the first inference on the data set by applying the baseline operational parameters to execution of the machine learning model. ### Claim 15 (depends on 13) The system of claim 13, wherein identifying operational parameters resulting in inference performance approaching the performance metric targets based on the machine learning model and operational properties of the computing device comprises: generating a plurality of sets of operational parameters, each set being associated with a different performance metric; and selecting one of the plurality of sets of operational parameters having a performance metric closest to the performance metric targets. ### Claim 16 (depends on 13) The system of claim 13, wherein the operation further comprises: detecting a change in the operational properties of the computing device; updating the operational parameters based on the machine learning model and the changed operational properties of the computing device; and applying the updated operational parameters to performance of additional inferences using the machine learning model. ### Claim 17 (depends on 13) The system of claim 13, wherein identifying operational parameters resulting in inference performance approaching the performance metric targets based on the machine learning model and operational properties of the computing device comprises inputting information about the machine learning model and the operational properties of the computing device into the operational parameter optimizer model trained using a training data set of performance data from previous inferences and performance metrics for the previous inferences. ### Claim 18 (depends on 17) The system of claim 17, wherein the operation further comprises: for each inference performed on the computing device, augmenting the training data set with performance data for each inference and operational parameters applied to the machine learning model for each inference; and periodically retraining the operational parameter optimizer model using the augmented training data set. ### Claim 19 (depends on 13) The system of claim 13, wherein identifying operational parameters resulting in inference performance approaching the performance metric targets based on the machine learning model and operational properties of the computing device comprises: transmitting, to the operational parameter optimizer model executing on a remote system, information about the machine learning model and the operational properties of the computing device; and receiving the operational parameters from the operational parameter optimizer model executing on the remote system. ### Claim 20 A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors configured individually or in any combination, performs an operation for performing inferences using a machine learning model on a computing device, the operation comprising: receiving a request to perform inferences on a data set using a machine learning model configured to perform the inferences and performance metric targets for performance of the inferences; performing a first inference on the data set using the machine learning model to meet a latency specified for generation of the first inference from receipt of the request, wherein the first inference is performed using baseline operational parameters configured for execution of inference operations using the machine learning model, wherein the baseline operational parameters comprise predefined parameters for the machine learning model; while performing the first inference using the machine learning model, simultaneously identifying operational parameters using an operational parameter optimizer model configured to identify the operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets based on the machine learning model and operational properties of a computing device; and applying the identified operational parameters to performance of subsequent inferences using the machine learning model.