Reducing power consumption by hardware accelerator during generation and transmission of machine learning inferences

A hardware accelerator comprising:

• at least one compute unit;
• and
• a machine-readable medium storing instructions that, when executed by the at least one compute unit, cause the at least one compute unit to perform operations, the operations comprising:
• receiving a slice of input data at a time-step wherein the input data is an image including text in a particular language, wherein the slice of input data is determined based on structural limitations of the hardware accelerator;
• processing the input data using a machine learning model deployed on the hardware accelerator to compute a respective probability of a plurality of probabilities for each of a plurality of classes, wherein the plurality of classes comprises a plurality of characters in the particular language, wherein the respective probability for each class is a likelihood that content in the slice belongs to the class;
• and
• determining a preset number of highest probabilities for the slice of input data, wherein the preset number of highest probabilities for the slice are transmitted to a host processor.

A non-transitory computer program product storing instructions that, when executed by at least one compute unit, cause the at least one compute unit to perform operations, the operations comprising:

• receiving a slice of input data at a time-step, wherein the input data is an image including text in a particular language, wherein the slice of input data is determined based on structural limitations of the hardware accelerator;
• processing the input data using a machine learning model deployed on the hardware accelerator to compute a respective probability of a plurality of probabilities for each of a plurality of classes, wherein the plurality of classes comprises a plurality of characters in the particular language, wherein the respective probability for each class is a likelihood that content in the slice belongs to the class;
• and
• determining a preset number of highest probabilities for the slice of input data, wherein the preset number of highest probabilities for the slice are to be transmitted to a host processor.