Mixed-precision neural networks

A system comprising:

• a processor;
• and
• a memory storing instructions, which when executed by the processor, cause the processor to perform operations comprising:
• receiving a target bandwidth increase for a neural network comprising a plurality of binary large objects (BLOBs) and weights of a first data type represented by a first number of bits, wherein the target bandwidth increase relates to changing a first portion of the plurality of BLOBs and weights to a second data type represented by a second number of bits different from the first number of bits and changing a second portion of the plurality of BLOBs and weights to a third data type represented by a third number of bits different from both the first and second numbers of bits;
• identifying both:
• (i) the first portion of the plurality of BLOBs and weights in the neural network to change from the first data type to the second data type and (ii) the second portion of the plurality of BLOBs and weights in the neural network to change from the first data type to the third data type, based on the target bandwidth increase, comprising:
• identifying, for a first BLOB or weight and a second BLOB or weight of the plurality of BLOBs and weights in the neural network, a total bandwidth of all instances of the first BLOB or weight in the neural network;
• and
• selecting both the first portion and the second portion, from among the plurality of BLOBs and weights, based on maintaining a total bandwidth for the neural network at or below the target bandwidth increase and the identified first BLOB or weight and second BLOB or weight;
• and
• changing, based on the selecting both the first portion and the second portion, the first portion of the plurality of BLOBs and weights from the first data type to the second data type and the second portion of the plurality of BLOBs and weights from the first data type to the third data type.

A non-transitory computer readable medium comprising stored instructions, which when executed by a processor, cause the processor to perform operations comprising:

• receiving a target performance change for a machine learning (ML) model comprising a plurality of objects of a first data type represented by a first number of bits, wherein the target performance change relates to changing a first portion of the plurality of objects to a second data type represented by a second number of bits different from the first number of bits and changing a second portion of the plurality of objects to a third data type represented by a third number of bits different from both the first and second numbers of bits;
• sorting the plurality of objects in the ML model based on size, comprising:
• identifying, for a first object of the plurality of objects in the ML model, a total size of all instances of the first object in the ML model;
• identifying both:
• (i) the a first portion of the plurality of objects in the ML model to change from the first data type to the second data type and (ii) the second portion of the plurality of objects in the ML model to change from the first data type to the third data type, based on the target performance change and the sorting the plurality of objects, comprising:
• selecting both the first portion and the second portion, from among the plurality of objects, based on maintaining performance for the ML model to meet the target performance change and using the sorted plurality of objects;
• changing, based on the selecting both the first portion and the second portion, the first portion of the plurality of objects from the first data type to the second data type and the second portion of the plurality of objects from the first data type to the third data type;
• and
• using the ML model, comprising the first portion of the plurality of objects changed to the second data type and the second portion of the plurality of objects changed to the third data type, to infer a result from one or more inputs.