Hardware node having a matrix vector unit with block-floating point processing

A system comprising:

• an input message processor configured to process incoming messages, wherein the input message processor is further configured to split the incoming messages into a first set of messages and a second set of messages;
• a scalar processor configured to process the first set of messages;
• and
• a neural function unit configured to process instructions placed in a plurality of queues by the scalar processor on input data received at least via the second set of messages, the neural function unit comprising:
• a pipeline configured to process instructions, the pipeline including a matrix vector unit, a first multifunction unit, wherein the first multifunction unit is connected to receive an input from the matrix vector unit, a second multifunction unit, wherein the second multifunction unit is connected to receive an output from the first multifunction unit, and a third multifunction unit, wherein the third multifunction unit is connected to receive an output from the second multifunction unit, and wherein the matrix vector unit is further configured to:
• receive training vector data corresponding to at least one layer of a neural network model for processing using the matrix vector unit, wherein each of the training vector data has a value represented in a format comprising at least one exponent, and
• first process a first subset of the training vector data to determine a first shared exponent for representing values in the first subset of the training vector data in a block-floating point format and second process a second subset of the training vector data to determine a second shared exponent for representing values in the second subset of the training vector data in a block-floating point format, wherein the processor is further configured to determine the first shared exponent and the second shared exponent in a manner that no vector data from the second subset of the training vector data influences a determination of the first shared exponent and no vector data from the first subset of the training vector data influences a determination of the second shared exponent.