# US 12,182,619 B2 · Annotation override determination for a neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,619 B2 - Document ID: US12182619B2 - Status: Granted - Kind: B2 - Application number: 18/074,440 - Filed: 2022-12-02 - Published: 2023-06-08 - Granted: 2024-12-31 - Earliest claimed: 2017-06-03 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Francesco Rossi; Gaurav Kapoor; Michael R. Siracusa; William B. March - Web page: https://patentgrep.com/patent/US12182619B2 ## CPC - G06F 9/50: Allocation of resources, e.g. of the central processing unit [CPU] - G06F 8/451: {Code distribution (considering CPU load at run-time G06F9/505; load rebalancing G06F9/5083)} - G06F 9/485: {Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence)} - G06F 9/5038: {considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups)} - G06F 9/5044: {considering hardware capabilities} - G06N 3/02: Neural networks - G06N 3/063: using electronic means - G06F 8/41: Compilation - G06F 2209/509: Offload - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The subject technology provides for dynamic task allocation for neural network models. The subject technology determines an operation performed at a node of a neural network model. The subject technology assigns an annotation to indicate whether the operation is better performed on a CPU or a GPU based at least in part on hardware capabilities of a target platform. The subject technology determines whether the neural network model includes a second layer. The subject technology, in response to determining that the neural network model includes a second layer, for each node of the second layer of the neural network model, determines a second operation performed at the node. Further the subject technology assigns a second annotation to indicate whether the second operation is better performed on the CPU or the GPU based at least in part on the hardware capabilities of the target platform. ## Claims ### Claim 1 A system comprising; a first processor; a second processor that has different hardware capabilities than the first processor; and a memory device containing instructions, which when executed by the first processor cause the first processor to: at runtime corresponding to execution of a neural network model, for each respective layer of a plurality of layers of the neural network model: determine an operation performed at a node of the respective a first-layer of the neural network model; determine whether to override an annotation of the operation performed at the node based at least in part on a current condition of the system, the annotation having been assigned prior to runtime based at least in part on a cost of transferring the operation between the first and second processor or a number of cycles for completing the operation on at least one of the first or second processor, the annotation indicating the first or second processor for performing the operation; and cause the operation to be executed in accordance with the determination of whether to override the annotation. ### Claim 2 (depends on 1) The system of claim 1, wherein the first processor comprises a CPU and the second processor comprises a GPU. ### Claim 3 (depends on 1) The system of claim 1, wherein to determine whether to override the annotation of the operation performed at the node further comprises: determining whether the first or second processor supports the operation. ### Claim 4 (depends on 3) The system of claim 3, wherein determining whether the first or second processor supports the operation is based on a registry of operations. ### Claim 5 (depends on 1) The system of claim 1, wherein to determine whether to override the annotation of the operation performed at the node is based on the cost of transferring the operation between the first and second processor. ### Claim 6 (depends on 1) The system of claim 1, wherein to determine whether to override the annotation of the operation performed at the node is based on a current utilization of the first and second processors. ### Claim 7 (depends on 1) The system of claim 1, wherein to determine whether to override the annotation of the operation performed at the node is based on a priority of an application that is running the neural network model. ### Claim 8 (depends on 1) The system of claim 1, wherein to determine whether to override the annotation of the operation performed at the node is based on a first priority of the first processor and a second priority of the second processor. ### Claim 9 (depends on 1) The system of claim 1, wherein to determine whether to override the annotation of the operation performed at the node is based on a current power usage or remaining battery level of the system. ### Claim 10 (depends on 1) The system of claim 1, wherein the respective first layer comprises an output layer. ### Claim 11 (depends on 1) The system of claim 1, wherein a same annotation is utilized for all nodes of the respective layer. ### Claim 12 A method comprising: at runtime corresponding to execution of a neural network model on a system, for each respective layer of a plurality of layers of the neural network model: determining an operation performed at a node of a the respective layer of the neural network model; determining whether to override an annotation of the operation performed at the node based at least in part on a current condition of the system, the annotation having been assigned prior to runtime based at least in part on a cost of transferring the operation between a first and second processor or a number of cycles for completing the operation on at least one of the first or second processor, wherein the first processor has different hardware capabilities than the second processor, and the annotation indicates the first or second processor for performing the operation; and cause the operation to be executed in accordance with the determining whether to override the annotation. ### Claim 13 (depends on 12) The method of claim 12, wherein the first processor comprises a CPU and the second processor comprises a GPU. ### Claim 14 (depends on 12) The method of claim 12, wherein determining whether to override the annotation of the operation performed at the node further comprises: determining whether the first or second processor supports the operation. ### Claim 15 (depends on 14) The method of claim 14, wherein determining whether the first or second processor supports the operation is based on a registry of operations. ### Claim 16 (depends on 12) The method of claim 12, wherein determining whether to override the annotation of the operation performed at the node is based on the cost of transferring the operation between the first and second processor. ### Claim 17 (depends on 12) The method of claim 12, wherein determining whether to override the annotation of the operation performed at the node is based on a current utilization of the first and second processor. ### Claim 18 (depends on 12) The method of claim 12, wherein determining whether to override the annotation of the operation performed at the node is based on a priority of an application that is running the neural network model. ### Claim 19 (depends on 12) The method of claim 12, wherein determining whether to override the annotation of the operation performed at the node is based on a first priority of the first processor and a second priority of the second processor. ### Claim 20 (depends on 12) The method of claim 12, wherein the respective layer comprises an output layer. ### Claim 21 (depends on 12) The method of claim 12, wherein a same annotation is utilized for all nodes of the respective layer. ### Claim 22 A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising: at runtime corresponding to execution of a neural network model, for each respective layer of a plurality of layers of the neural network model: determining an operation performed at a node of the respective layer of the neural network model; determining whether to override an annotation of the operation performed at the node based at least in part on a current condition of the computing device, the annotation indicating whether the operation is suggested to be performed on a first processor or a second processor having different hardware capabilities than the first processor, wherein the annotation having been assigned based at least in part on a cost of performing the operation at the first or second processor or a cost of transferring the operation between the first and second processor; and causing the operation to be executed in accordance with the determining whether to override the annotation. ### Claim 23 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein a same annotation is utilized for all nodes of the respective layer.