Loading patent…
AssigneeThe Toronto-Dominion Bank
InventorsPeter Starszyk, Wen-Yang Ku, Guangwei Yu, Devinder Kumar, Maksims Volkovs
Estimating the term…
AI models such as boosted decision trees, neural networks, large language models (LLMs), and agentic systems are deployed with multiple multi-threaded workers to service requests in parallel. The processing configuration is analyzed to determine distance between thread execution units (e.g., vCPUs) and a mapping determined to optimize thread assignment to the thread execution units. The threads may be statically pinned to particular vCPUs to prevent thread migration and optimize throughput. The thread mapping accounts for thread workloads and core-to-core distance. The mapping may seek to maximize distance between worker main threads and may either minimize (compact) or maximize (spread) distance between threads within the same worker. The mapping may be determined as an optimization problem, such as an Integer Linear Problem (ILP) that defines an objective and various constraints for optimization.
A system for managing execution of model inference workers, comprising:
A computer-implemented method for managing execution of model inference workers, comprising:
A non-transitory computer-readable medium for managing execution of model inference workers, the non-transitory computer-readable medium comprising instructions executable by a processor for:
Dependencies are inferred from the claim text.