# US 12,182,688 B2 · Hierarchical partitioning of operators > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,688 B2 - Document ID: US12182688B2 - Status: Granted - Kind: B2 - Application number: 16/698,236 - Filed: 2019-11-27 - Published: 2021-05-27 - Granted: 2024-12-31 - Earliest claimed: 2019-11-27 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Animesh Jain; Yizhi Liu; Hongbin Zheng; Jeffrey T. Huynh; Haichen Li; Drazen Borkovic; Jindrich Zejda; Richard John Heaton; Randy Renfu Huang; Zhi Chen; Yida Wang - Web page: https://patentgrep.com/patent/US12182688B2 ## CPC - G06N 3/063: using electronic means - G06N 3/105: {Shells for specifying net layout} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06F 8/4441: {Reducing the execution time required by the program code} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and apparatuses for hierarchical partitioning of operators of a neural network for execution on an acceleration engine are provided. Neural networks are built in machine learning frameworks using neural network operators. The neural network operators are compiled into executable code for the acceleration engine. Development of new framework-level operators can exceed the capability to map the newly developed framework-level operators onto the acceleration engine. To enable neural networks to be executed on an acceleration engine, hierarchical partitioning can be used to partition the operators of the neural network. The hierarchical partitioning can identify operators that are supported by a compiler for execution on the acceleration engine, operators to be compiled for execution on a host processor, and operators to be executed on the machine learning framework. ## Claims ### Claim 1 A method for hierarchical partitioning of operators of a neural network for compiling on an acceleration engine, the method comprising: executing a machine learning framework to generate a set of neural network operators, wherein at least some of the neural network operators are executable on the acceleration engine; comparing a list of neural network operators supported by a compiler with the set of neural network operators of the neural network; partitioning, by a partitioner based on the comparison, the set of neural network operators of the neural network into a set of first neural network operators that are not supported by the compiler and a set of second neural network operators that are supported by the compiler; providing the set of second neural network operators to the compiler; partitioning, by the partitioner, the set of second neural network operators into a set of third neural network operators initially determined to be capable of execution on the acceleration engine and a set of fourth neural network operators to be executed by a host processor; evaluating parameters of the set of third neural network operators; determining that at least some of the neural network operators in the set of third neural network operators initially determined to be capable of execution on the acceleration engine include parameters that are not supported by the acceleration engine; partitioning, by the partitioner, the set of third neural network operators into a set of fifth neural network operators to be executed on the host processor, the set of fifth neural network operators comprising specific instances of the neural network operators having the parameters that are not supported by the acceleration engine; compiling a first model of a first subset of the neural network operators supported by the compiler and having parameters that are supported by the acceleration engine for execution on the acceleration engine; and compiling a second model of a second subset of the operators supported by the compiler for execution on the host processor, the second subset of the operators including the specific instances of the neural network operators having the parameters that are not supported by the acceleration engine. ### Claim 2 (depends on 1) The method of claim 1, further comprising compiling the neural network operators of the third set of neural network operators having parameters that are supported by the acceleration engine for execution on the acceleration engine. ### Claim 3 (depends on 1) The method of claim 1, further comprising compiling the fourth set of neural network operators for execution on the host processor. ### Claim 4 A method, comprising: executing a machine learning framework to generate operators of a neural network, wherein at least some of the neural network operators are executable on an acceleration engine; identifying neural network operators from the neural network that are supported by a compiler for an acceleration engine; partitioning, by a partitioner, the neural network operators that are supported by the compiler into a set of first operators that is initially determined to be executable by the acceleration engine and a set of second operators that is executable by a host processor; evaluating parameters of the first set of operators; determining that specific instances of neural network operators of the first set of operators include a parameter that is not supported by the acceleration engine; partitioning, by the partitioner, the operator that includes the parameter that is not supported by the acceleration engine for execution on the host processor; compiling a first model of a first subset of the operators supported by the compiler for execution on the acceleration engine; and compiling a second model of a second subset of the operators supported by the compiler for execution on the host processor, the second subset of the operators including the specific instances of neural network operators having the parameters that are not supported by the acceleration engine. ### Claim 5 (depends on 4) The method of claim 4, wherein the identifying comprises: receiving a list of neural network operators that are supported by the compiler; and comparing the neural network operators from the neural network to the list. ### Claim 6 (depends on 5) The method of claim 5, further comprising partitioning the neural network operators that are not on the list of neural network operators that are supported by the compiler for execution on the machine learning framework. ### Claim 7 (depends on 4) The method of claim 4, wherein the partitioning comprises: determining that an operator of the first set of operators is supported by the acceleration engine; determining that the operator can execute faster on the host processor than on the acceleration engine; and partitioning the operator for execution on the host processor. ### Claim 8 (depends on 4) The method of claim 4, wherein the partitioning comprises: evaluating a sequence of operators of the set of first operators; determining that the sequence of operators is not supported by the acceleration engine; and partitioning the sequence of operators for execution on the host processor. ### Claim 9 (depends on 4) The method of claim 4, further comprising: determining that compiling the set of first operators failed; and repartitioning the first set of operators for execution on the machine learning framework. ### Claim 10 A system, comprising: a set of processors; and an acceleration engine, wherein the set of processors is configured to: execute a machine learning framework to generate neural network operators, wherein at least some of the operators are executable on the acceleration engine; partition, by a partitioner, the neural network operators into a first subset of operators that are initially determined to be supported by a compiler for the acceleration engine and a set of second operators that is executable by the set of processors; evaluate parameters of the first subset of operators; determine that a parameter of an operator of the first subset of operators initially determined to be supported by a compiler for the acceleration engine is not supported by the acceleration engine; and partition, by the partitioner, the operator of the first subset of operators initially determined to be capable of execution supported by the compiler for the acceleration engine for execution on the set of processors; compile a first model of the first subset of the operators supported by the compiler for execution on the acceleration engine; and compile a second model of a second subset of the operators supported by the compiler for execution on the set of processors, the second subset of the operators including specific instances of neural network operators having the parameters that are not supported by the acceleration engine. ### Claim 11 (depends on 10) The system of claim 10, further comprising a partitioner configured to: determine which of the operators are not supported by the compiler; and partition for execution on the machine learning framework the operators not supported by the compiler. ### Claim 12 (depends on 11) The system of claim 11, wherein the partitioner is further configured to: receive, from the compiler, a list of operators supported by the compiler, compare the operators of the neural network to the list, and based on the comparison, partition the neural network operators supported by the compiler and operators not supported by the compiler. ### Claim 13 (depends on 10) The system of claim 10, wherein the set of processors is further configured to: evaluate parameters of an operator; and partition the operator for execution on the set of processors when a parameter that is not supported by the acceleration engine is detected. ### Claim 14 (depends on 10) The system of claim 10, wherein the set of processors is further configured to: evaluate a sequence of operators of the first subset of operators; determine that the sequence of operators is not supported by the acceleration engine; and partition the sequence of operators for execution on the set of processors. ### Claim 15 (depends on 10) The system of claim 10, wherein the acceleration engine is configured to execute the first compiled model. ### Claim 16 (depends on 10) The system of claim 10, wherein the set of processors is configured to execute the second compiled model. ### Claim 17 A non-transitory computer readable medium having stored therein instructions for making one or more processors execute a method for partitioning operators of a neural network, the processor executable instructions comprising instructions for performing operations including: executing a machine learning framework to generate operators of a neural network, wherein at least some of the neural network operators are executable on an acceleration engine; identifying neural network operators from the neural network that are supported by a compiler for an acceleration engine; partitioning, by a partitioner, the neural network operators that are supported by the compiler into a set of first operators that is initially determined to be executable by the acceleration engine and a set of second operators that is executable by a host processor; evaluating parameters of the first set of operators; determining that specific instances of neural network operators of the first set of operators include a parameter that is not supported by the acceleration engine; partitioning, by the partitioner, the operator for execution on the host processor; compiling a first model of a first subset of the operators supported by the compiler for execution on the acceleration engine; and compiling a second model of a second subset of the operators supported by the compiler for execution on the host processor, the second subset of the operators including the specific instances of neural network operators having the parameters that are not supported by the acceleration engine.