Instruction set architecture (ISA) format for multiple instruction set architectures in machine learning inference engine

A programmable hardware system for machine learning (ML), comprising:

• an instruction streaming engine configured to
• receive a set of commands in a first instruction set architecture (ISA) format from a core,
• convert the set of commands from the first ISA format into a second ISA format, wherein the set of commands in the first ISA format is an asynchronous instruction set, wherein each command of the set of commands in the first ISA format programs a state machine independent of other state machines and runs asynchronously,
• stream the set of commands in the second ISA format to an inference engine;
• and
• said inference engine configured to
• receive the set of commands in the second ISA format,
• execute the set of commands in the second ISA format to program a plurality of components within the inference engine to perform an ML operation.

A method comprising:

• receiving a set of commands for performance-critical operations in a first instruction set architecture (ISA) format from a core;
• generating a second ISA format from the first ISA format;
• streaming the set of commands in the second ISA format to an inference engine, wherein the set of commands in the first ISA format is an asynchronous instruction set;
• and
• executing by the inference engine the stream of the set of commands in the second ISA format to program a plurality of components within the inference engine to perform an ML operation.

A programmable hardware system for machine learning (ML), comprising:

• a core configured to
• receive a plurality of commands from a host, wherein the plurality of commands is in a first instruction set architecture (ISA) format to coordinate components in the programmable hardware system to perform an ML operation;
• divide the plurality of commands into a first set of commands and a second set of commands for performing a first set of operations and a second set of operations of the ML operation, respectively;
• execute the second set of commands to perform the second set of operations of the ML operation without executing the first set of commands;
• and
• write the first set of commands and data associated therewith into a memory component, via a function call, for use by an inference engine,
• wherein the memory component for storing the first set of commands and the associated data is a circular buffer, wherein a first pointer is used to track writing to the circular buffer and wherein a second pointer is used to track reading from the circular buffer, wherein the core is configured to maintain an available buffer size of the circular buffer and wherein the core is configured to decrement the available buffer size for every write, wherein the core is configured to continue writing to the circular buffer if the available buffer size exceeds a threshold value, and wherein the core is configured to read the second pointer from a memory mapped space if the available buffer size is less than the threshold value and the core is further configured to reset the available buffer size.

A method comprising:

• receiving a plurality of commands from a host, wherein the plurality of commands is in a first instruction set architecture (ISA) format to coordinate components in the programmable hardware system to perform an ML operation;
• dividing the plurality of commands into a first set of commands and a second set of commands for performing a first set of operations and a second set of operations of the ML operation, respectively;
• executing the second set of commands to perform the second set of operations of the ML operation without executing the first set of commands;
• writing the first set of commands and data associated therewith into a memory component, via a function call, for use by an inference engine, wherein the memory component is a circular buffer;
• using a first pointer to track writing to the circular buffer and using a second pointer to track reading from the circular buffer;
• maintaining an available buffer size of the circular buffer;
• decrementing the available buffer size for every write;
• continue writing to the circular buffer if the available buffer size exceeds a threshold value;
• reading the second pointer from a memory mapped space if the available buffer size is less than the threshold value;
• and
• resetting the available buffer size after reading the second pointer.

A programmable hardware system for machine learning (ML), comprising:

• an instruction streaming engine configured to
• receive a set of commands in a first instruction set architecture (ISA) format from a core,
• convert the set of commands from the first ISA format into a second ISA format,
• stream the set of commands in the second ISA format to an inference engine;
• and
• said inference engine configured to
• receive the set of commands in the second ISA format,
• execute the set of commands in the second ISA format to program a plurality of components within the inference engine to perform an ML operation, wherein the plurality of components within the inference engine comprises a plurality of first processing units (PODs) and a plurality of second processing units/elements (PEs), wherein each POD of the plurality of PODs and each PE of the plurality of PEs is coupled to one on-chip memory (OCM), and wherein each POD is configured to perform a dense and/or regular computation task of the ML operation and wherein each PE is configured to perform a sparse and/or irregular computation task of the ML operation.

A programmable hardware system for machine learning (ML), comprising:

• a core configured to
• receive a plurality of commands from a host, wherein the plurality of commands is in a first instruction set architecture (ISA) format to coordinate components in the programmable hardware system to perform an ML operation;
• divide the plurality of commands into a first set of commands for performance-critical operations and a second set of commands for performance non-critical operations of the ML operation;
• execute the second set of commands to perform the performance non-critical operations of the ML operation without executing the first set of commands for the performance-critical operations;
• and
• provide the first set of commands for the performance-critical operations to be executed by an inference engine, and wherein the inference engine is configured to execute commands in a second ISA format,
• wherein each command of the first set of commands in the first ISA format programs a state machine independent of other state machines and runs asynchronously.

A programmable hardware system for machine learning (ML), comprising:

• an instruction streaming engine configured to
• receive a set of commands in a first instruction set architecture (ISA) format from a core, wherein each command of the set of commands in the first ISA format programs a state machine independent of other state machines and runs asynchronously,
• convert the set of commands from the first ISA format into a second ISA format,
• stream the set of commands in the second ISA format to an inference engine;
• and
• said inference engine configured to
• receive the set of commands in the second ISA format,
• execute the set of commands in the second ISA format to program a plurality of components within the inference engine to perform an ML operation.

A method comprising:

• receiving a set of commands for performance-critical operations in a first instruction set architecture (ISA) format from a core;
• generating a second ISA format from the first ISA format;
• streaming the set of commands in the second ISA format to an inference engine, wherein each command of the set of commands in the first ISA format programs a state machine independent of other state machines and runs asynchronously;
• and
• executing by the inference engine the stream of the set of commands in the second ISA format to program a plurality of components within the inference engine to perform an ML operation.