# US 2022/0035684 A1 · Dynamic load balancing of operations for real-time deep learning analytics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0035684 A1 - Document ID: US20220035684A1 - Status: Published application - Kind: A1 - Application number: 17/330,710 - Filed: 2021-05-26 - Published: 2022-02-03 - Earliest claimed: 2020-08-03 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Shaunak Gupte; Amit Kale; Bhushan Rupde; Kaustubh Purandare - Web page: https://patentgrep.com/patent/US20220035684A1 ## CPC - G06F 9/5083: {Techniques for rebalancing the load in a distributed system} - G06F 9/5088: {involving task migration} - G06F 9/30079: {Pipeline control instructions, e.g. multicycle NOP} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06F 2209/5022: Workload threshold - G06F 2209/506: Constraint - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Apparatuses, systems, and techniques to balance processing load between a plurality of hardware accelerators. In at least one embodiment, operations performed on batches of frames of a video (e.g., as part of a video analytics pipeline) are distributed by a load balancer between a first hardware accelerator and a second hardware accelerator. ## Claims ### Claim 1 A system comprising: one or more processors; and memory storing instructions that, as a result of being executed by the one or more processors, cause the system to: assign a set of clients to a first hardware accelerator, where a first client of the set of clients provides a batch of frames for processing; generate a first determination that a first metric associated with processing the batch of frames using the first hardware accelerator exceeds a threshold; and assign a subset of clients of the set of clients to a second hardware accelerator in response to the first determination. ### Claim 2 (depends on 1) The system of claim 1, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to: generate a second determination that the first metric associated with the first hardware accelerator is below the threshold; and assign the subset of clients of the set of clients to the first hardware accelerator in response to the second determination. ### Claim 3 (depends on 1) The system of claim 1, wherein the first metric further comprises a value indicating a percentage of activity of the first hardware accelerator. ### Claim 4 (depends on 1) The system of claim 1, wherein the first hardware accelerator further comprises a video image compositor (VIC). ### Claim 5 (depends on 1) The system of claim 1, wherein the second hardware accelerator further comprises a graphics processing unit (GPU). ### Claim 6 (depends on 1) The system of claim 1, wherein the first metric further comprises an amount of time the first client utilizes the first hardware accelerator to perform processing of the batch of frames. ### Claim 7 (depends on 1) The system of claim 1, wherein the first metric further comprises a percentage of processing capability the first hardware accelerator utilized to process the batch of frames on behalf of the first client. ### Claim 8 (depends on 1) The system of claim 1, wherein the first metric further comprises an average amount of load generated by at least processing the batch of frames provided by the first client. ### Claim 9 (depends on 1) The system of claim 1, wherein the first determination is generated during an interval of time. ### Claim 10 (depends on 1) The system of claim 1, wherein the first determination is generated based at least in part on historical data. ### Claim 11 (depends on 1) The system of claim 1, wherein the instructions that cause the system to assign the subset of clients of the set of clients to the second hardware accelerator further include instructions that, as a result of being executed by the one or more processors, cause the system to assign the subset of clients of the set of clients to the second hardware accelerator based at least in part on a preference provided by a user. ### Claim 12 (depends on 1) The system of claim 1, wherein the threshold is designated by a user. ### Claim 13 (depends on 1) The system of claim 1, wherein a user indicates a preference between the first hardware accelerator and the second hardware accelerator for processing on behalf of at least one client of the set of clients. ### Claim 14 (depends on 1) The system of claim 1, wherein the first metric is obtained from a hardware performance counter included in the first hardware accelerator. ### Claim 15 (depends on 1) The system of claim 1, wherein the first hardware accelerator further comprises a field-programmable gate array (FPGA). ### Claim 16 (depends on 1) The system of claim 1, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to obtain, through a system call, the first metric. ### Claim 17 (depends on 1) The system of claim 1, wherein the set of clients further comprise a set of components of an artificial intelligence pipeline. ### Claim 18 (depends on 17) The system of claim 17, wherein the artificial intelligence pipeline includes one or more neural networks. ### Claim 19 (depends on 1) The system of claim 1, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to cause the first hardware accelerator to process the batch of frames by at least converting the batch of frames from a first format to a second format. ### Claim 20 (depends on 1) The system of claim 1, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to cause the first hardware accelerator to process the batch of frames by at least scaling the batch of frames. ### Claim 21 (depends on 1) The system of claim 1, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to cause the first hardware accelerator to process the batch of frames by at least modifying one or more color values associated with at least one frame of the batch of frames. ### Claim 22 A method comprising: assigning one or more application clients to a video image compositor (VIC) engine; comparing an average time used by the VIC engine to perform processing for the one or more application clients to a frame processing threshold; load balancing, to a processing unit, a processing load of an application client of the one or more application clients based at least in part on the average time used by the VIC engine; determining an average time used by the processing unit to perform processing for the application client is lower than at least one other application client; and moving the processing load of the application client from the processing unit back to the VIC. ### Claim 23 (depends on 22) The method of claim 22, wherein the determining and the moving are repeated one or more times. ### Claim 24 (depends on 22) The method of claim 22, wherein the frame processing threshold is determined based at least in part on a framerate of video processing. ### Claim 25 (depends on 22) The method of claim 22, wherein the load balancing is executed using a load balancer. ### Claim 26 (depends on 25) The method of claim 25, wherein the load balancer comprises a processing thread. ### Claim 27 (depends on 25) The method of claim 25, wherein the load balancer maintains a table of application clients. ### Claim 28 (depends on 27) The method of claim 27, wherein the table of application clients includes a processing engine assigned to each client and an average time taken by the client to compute transformations for each processing engine. ### Claim 29 (depends on 22) The method of claim 22, wherein the processing unit comprises a GPU. ### Claim 30 A non-transitory computer readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to: obtain performance data associated with a video image compositor (VIC), the performance data related to an application client assigned to the VIC; determine, based at least in part on the performance data, that one or more conditions are satisfied; and as a result of the one or more conditions being satisfied, assign the application client to a second processor to cause the second processor to relieve load from the VIC. ### Claim 31 (depends on 30) The non-transitory computer readable storage medium of claim 30, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to reassign the application client to the VIC. ### Claim 32 (depends on 30) The non-transitory computer readable storage medium of claim 30, wherein the second processor further comprises a field-programmable gate array (FPGA). ### Claim 33 (depends on 30) The non-transitory computer readable storage medium of claim 30, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to maintain a table of application clients of which the application client is a member. ### Claim 34 (depends on 33) The non-transitory computer readable storage medium of claim 33, wherein the table of application clients includes information indicating that the VIC or the second processor is assigned to the application clients and an average time taken by the VIC or the second processor to compute transformations for the application clients.