Learning quantization levels

A system for reducing data transmission for a plurality of local neural networks trained using distributed or federated learning, the system comprising:

• a plurality of computing devices respectively training the plurality of local neural networks, including a first computing device having a first memory that includes a first local neural network of the plurality of local neural networks, and a second computing device having a second memory that includes a second local neural network of the plurality of local neural networks;
• and
• a network connecting the plurality of computing devices,
• wherein the system is configured to:
• train a quantization neural network that is separate from the plurality of local neural networks, by using a plurality of training vectors each having a dimension k, the quantization neural network being configured to output quantization levels based on said training of the quantization neural network, each of the quantization levels being a value to which vector coordinates may be approximated, and said training of the quantization neural network including backpropagating, through the quantization neural network, errors between outputs generated by the quantization neural network during said training of the quantization neural network and expected quantization levels;
• after said training of the quantization neural network, randomly sample coordinates within a gradient vector having a dimension d, to obtain a first set of k coordinates, d being greater than k, and the gradient vector including gradient values generated by the first computing device by training the first local neural network;
• input the first set of k coordinates to the quantization neural network, and obtain first quantization levels as outputs generated by the quantization neural network based on the input first set of k coordinates;
• quantize the gradient vector based on the determined first quantization levels to generate a quantized gradient vector, coordinates within the quantized gradient vector including different quantization levels from the first quantization levels;
• and
• use the quantized gradient vector with the different quantization levels to train at least one of the plurality of local neural networks.

One or more non-transitory computer-readable media comprising instructions executable in a computer system, the computer system including a plurality of computing devices respectively training a plurality of local neural networks using distributed or federated learning, wherein the instructions, when executed, cause the computer system to carry out a method of reducing data transmission for the plurality of local neural networks, the method comprising:

• training a quantization neural network that is separate from the plurality of local neural networks, by using a plurality of training vectors each having a dimension k, the quantization neural network being configured to output quantization levels based on said training of the quantization neural network, each of the quantization levels being a value to which vector coordinates may be approximated, and said training of the quantization neural network including backpropagating, through the quantization neural network, errors between outputs generated by the quantization neural network during said training of the quantization neural network and expected quantization levels;
• after said training of the quantization neural network, randomly sampling coordinates within a gradient vector having a dimension d, to obtain a first set of k coordinates, d being greater than k, and the gradient vector including gradient values generated by a first computing device of the plurality of computing devices by training a first local neural network of the plurality of local neural networks;
• inputting the first set of k coordinates to the quantization neural network, and obtaining first quantization levels as outputs generated by the quantization neural network based on the input first set of k coordinates;
• quantizing the gradient vector based on the determined first quantization levels to generate a quantized gradient vector, coordinates within the quantized gradient vector including different quantization levels from the first quantization levels;
• and
• using the quantized gradient vector with the different quantization levels to train at least one of the plurality of local neural networks.