Architecture for irregular operations in machine learning inference engine

A processing unit of an inference engine for machine learning (ML), comprising:

• a first data load streamer configured to stream a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
• a second data load streamer configured to stream a second data stream comprising a second plurality of data sections from the OCM, using a single instruction, to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
• the operator component configured to perform a matrix operation on the first data stream and the second data stream, wherein the matrix operation is performed by another processing unit that reads data within each matrix only once and wherein the another processing unit is configured to receive data within the each matrix as a data stream using a single instruction and further configured to operate on the each matrix as the data stream using a single instruction to generate an output matrix;
• and
• a store streamer configured to receive a data output stream from the operator component and to store the data output stream in a buffer,
• wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
• wherein the data output stream is specified by a starting address, a number of lines to write, line stride between lines, a number of bytes per line, and stride to a next block.

A method comprising:

• streaming a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
• streaming a second data stream comprising a second plurality of data sections from the OCM to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
• performing a data operation on the first data stream and the second data stream;
• streaming a data output stream resulting from the performing;
• and
• storing the data output stream,
• wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
• wherein the first data stream pattern is specified by a starting address, number of lines to read for each operation, number of bytes per line, and a number of blocks to read.

A method comprising:

• streaming a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
• streaming a second data stream comprising a second plurality of data sections from the OCM to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
• performing a data operation on the first data stream and the second data stream;
• streaming a data output stream resulting from the performing;
• and
• storing the data output stream,
• wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
• wherein the data output stream is specified by a starting address, a number of lines to write, line stride between lines, a number of bytes per line, and stride to a next block.