Low latency long short-term memory inference with sequence interleaving

A method comprising:

• detecting, by a processing unit comprising circuitry, two or more sequences, stored in a memory device, that will be processed by a machine learning engine;
• interleaving, by the processing unit, the two or more sequences together into one or more data blocks, such that each data block comprises samples from the two or more sequences;
• providing the one or more data blocks to the machine learning engine;
• receiving, by the machine learning engine, a given data block of the one or more data blocks;
• performing, by circuitry of a plurality of matrix multiplication units of the machine learning engine, a plurality of matrix multiplication operations in parallel on a plurality of sequences from the given data block and a plurality of coefficients to generate outputs;
• and
• conveying, by the machine learning engine, the outputs from the plurality of matrix multiplication units to one or more long short-term memory (LSTM) layers.

An apparatus comprising:

• a machine learning engine comprising circuitry of a plurality of matrix multiplication units;
• and
• a memory device coupled to the machine learning engine;
• wherein the machine learning engine comprises circuitry configured to:
• receive a given data block comprising a plurality of sequences interleaved together;
• perform, in parallel by circuitry of the matrix multiplication units, a plurality of matrix multiplication operations on the plurality of sequences from the given data block and a plurality of coefficients to generate outputs;
• and
• convey the outputs from the plurality of matrix multiplication units to one or more long short-term memory (LSTM) layers.