# US 2019/0244141 A1 · Systems and methods for programmable hardware architecture for machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0244141 A1 - Document ID: US20190244141A1 - Status: Published application - Kind: A1 - Application number: 16/186,313 - Filed: 2018-11-09 - Published: 2019-08-08 - Earliest claimed: 2018-02-08 - Assignees: Cavium, Inc. - Applicants: Cavium, LLC - Inventors: Avinash SODANI; Chia-Hsin CHEN; Ulf R. HANEBUTTE; Hamid Reza GHASEMI; Senad DURAKOVIC - Web page: https://patentgrep.com/patent/US20190244141A1 ## CPC - G06F 9/3879: {for non-native instruction execution, e.g. executing a command; for Java instruction set} - G06F 9/30174: {for non-native instruction set, e.g. Javabyte, legacy code} - G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution} - G06F 9/3851: {from multiple instruction streams, e.g. multistreaming} - G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)} - G06F 15/7807: {System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package} - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 20/00: Machine learning - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06F 9/3001: {Arithmetic instructions} - G06F 15/7864: {on more than one IC chip} - G06F 15/8023: {Two dimensional arrays, e.g. mesh, torus} - G06F 2212/602: Details relating to cache prefetching - G06N 5/04: Inference or reasoning models - G06N 20/20: Ensemble learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A programmable hardware architecture for machine learning (ML) is proposed, which includes at least a host, a memory, a core, a data streaming engine, a instruction-streaming engine, and an interference engine. The core interprets a plurality of ML commands for a ML operation and/or data received from the host and coordinate activities of the engines based on the data in the received ML commands. The instruction-streaming engine translates the ML commands received from the core and provide a set of programming instructions to the data streaming engine and the inference engines based on the translated parameters. The data steaming engine sends one or more data streams to the inference engine in response to the received programming instructions. The inference engine then processes the data streams received from the data stream engine according to the programming instructions received from the instruction-streaming engine. ## Claims ### Claim 1 A programmable hardware system for machine learning (ML), comprising: a memory configured to receive data from a host to be analyzed and/or inferred via machine learning; a core configured to receive a plurality of commands from the host and wherein the core is further configured to coordinate components in the system to perform a ML operation on the data, wherein said core is further configured to interpret and divide the plurality of commands received from the host into a first set of commands for dense operations, a second set of commands for irregular operations of the ML operation, and a third set of commands for operations other than dense and irregular operations, wherein the core is further configured to provide the first and the second set of commands via an instruction-streaming engine; a dense operation engine configured to execute the first set of commands to perform the dense operations of the ML operation; an irregular operation engine configured to execute the second set of commands to perform the irregular operations of the ML operation; a general processor configured to execute the third set of commands to perform the operations other than dense and irregular operations of the ML operation; said instruction-streaming engine coupled to the core, a data streaming engine, the dense operation engine, and the irregular operation engine, wherein the instruction-streaming engine is configured to stream a first set of programming instructions associated with the first and the second set of commands to the dense and the irregular operation engines based on the third set of commands; and said data steaming engine coupled to the dense and irregular operation engines and configured to generate one or more streams of data from the memory to be analyzed and/or inferred by the dense and irregular operation engines; and said dense and irregular operation engines configured to receive the data and the first and second set of programming instructions from the data streaming engine and the instruction-streaming engine, respectively; and perform the first and second set of commands by executing the first set of programming instructions to analyze the streams of data received from the data streaming engine. ### Claim 2 (depends on 1) The system of claim 1, wherein: the data to be analyzed and/or inferred is an image and the ML operation is to identify a subject in the image through inference. ### Claim 3 (depends on 1) The system of claim 1 further comprising a host configured to: generate the data to be analyzed and/or inferred and the plurality of commands; and provide training data to be used by the dense operation engine and further by the irregular operation engine for the ML operation, wherein the training data includes a polynomial with their respective weights. ### Claim 4 (depends on 3) The system of claim 3, wherein: the memory is configured to receive the data to be analyzed and/or the training data from a networking component via a direct memory access engine (DMA). ### Claim 5 (depends on 1) The system of claim 1, wherein: the core is the general processor and wherein the core is an FPGA, a CPU, or a microcontroller. ### Claim 6 (depends on 1) The system of claim 1, wherein: the core is configured to store the plurality of commands in a ML command RAM. ### Claim 7 (depends on 6) The system of claim 6, wherein: the ML command RAM is integrated with a networking component to reduce extra hops and accelerate access to the ML command RAM and/or the memory. ### Claim 8 (depends on 1) The system of claim 1, wherein: the performance non-critical operations include one or more of data and/or instruction preparation, data collection, and data mapping. ### Claim 9 (depends on 1) The system of claim 1, wherein: the performance critical operations include matrix multiplication. ### Claim 10 (depends on 1) The system of claim 1, wherein: the core is configured to allocate one or more locations in the memory for storing of one or more of the plurality of commands, the data, result after the data is processed to be accessed and used by the core or other components in the system. ### Claim 11 (depends on 1) The system of claim 1, wherein: the core, the dense and irregular operation engines are configured to execute the entire ML operation by themselves without relying on the host to execute certain ML commands. ### Claim 12 (depends on 1) The system of claim 1, wherein: the instruction-streaming engine is configured to provide a second set of programming instructions to the data streaming engine to stream the data to the dense and irregular operation engines to be analyzed and/or inferred. ### Claim 13 (depends on 12) The system of claim 12, wherein: the data streaming engine is configured to stream the data to be analyzed and/or inferred from the memory to the dense and irregular operation engines according to the second set of programming instructions from the instruction-streaming engine. ### Claim 14 A method to support machine learning (ML) via a programmable hardware system, comprising: receiving data from a host to be analyzed and/or inferred via machine learning; receiving a plurality of commands; coordinating components in the system to perform a ML operation on the data based on the plurality of commands; interpreting and dividing the plurality of commands into a first set of commands for performance-critical operations and a second set of commands for performance non-critical operations of the ML operation; executing the second set of commands on a core to perform the performance non-critical operations of the ML operation; streaming a first set of programming instructions to an inference engine based on the first set of commands; generating and transmitting one or more streams of data from a memory to be analyzed and/or inferred by the inference engine; and processing the one or more streams of data based on the first set of programming instructions, at the inference engine. ### Claim 15 (depends on 14) The method of claim 14, further comprising: generating the data to be analyzed and/or inferred and the plurality of commands; providing training data to be used by the inference engine for the ML operation, wherein the training data includes a polynomial with their respective weights. ### Claim 16 (depends on 15) The method of claim 15, further comprising: receiving the data to be analyzed and/or the training data from a networking component via a direct memory access engine (DMA). ### Claim 17 (depends on 14) The method of claim 14, further comprising: storing the plurality of commands in a ML command RAM. ### Claim 18 (depends on 17) The method of claim 17, further comprising: integrating the ML command RAM with a networking component to reduce extra hops and accelerate access to the ML command RAM and/or the memory. ### Claim 19 (depends on 14) The method of claim 14, further comprising: allocating one or more locations in the memory for storing of one or more of the plurality of commands, the data, and result after the data is processed to be accessed and used by the core or other components in the system. ### Claim 20 (depends on 14) The method of claim 14, further comprising: executing the entire ML operation by the core and the inference engine without relying on the host to execute certain ML commands. ### Claim 21 (depends on 14) The method of claim 14, further comprising: providing a second set of programming instructions to a data streaming engine to stream the data to the inference engine to be analyzed and/or inferred. ### Claim 22 (depends on 21) The method of claim 21, further comprising: streaming the one or more streams of the data to be analyzed and/or inferred from the memory to the inference engine according to the second set of programming instructions. ### Claim 23 A programmable hardware system for machine learning (ML), comprising: a memory configured to receive data from a host to be analyzed and/or inferred via machine learning; a core configured to receive a plurality of commands from the host and wherein the core is further configured to coordinate components in the system to perform a ML operation on the data, wherein said core is further configured to interpret and divide the plurality of commands received from the host into a first set of commands for performance-critical operations and a second set of commands for performance non-critical operations of the ML operation, and wherein the core is further configured to execute the second set of commands to perform the performance non-critical operations of the ML operation and wherein the core is further configured to provide the first set of commands for the performance-critical operations to be performed by an inference engine via an instruction-streaming engine; said instruction-streaming engine coupled to the core, a data streaming engine, and the inference engine, wherein the instruction-streaming engine is configured to stream a first set of programming instructions to the inference engine based on the second set of commands to perform the performance-critical operations; said data steaming engine coupled to the inference engine and configured to generate one or more streams of data from the memory to be analyzed and/or inferred by the inference engine; and said inference engine configured to receive the data and the first set of programming instructions from the data streaming engine and the instruction-streaming engine, respectively; perform the first set of commands for performance-critical operations of the ML operation by executing the first set of programming instructions to analyze the streams of data received from the data streaming engine.