Residual initialization for training node networks

A computer-implemented method for training a computational network, comprising:

• instantiating a node network comprising a plurality of nodes and connections, the plurality of nodes including a set of multi-input nodes each configured to receive at least two inputs and produce a node output;
• for each multi-input node in the set, providing a plurality of candidate discrete operations selectable for use in a discretized form of the node network, each candidate discrete operation being configured to map the at least two inputs of the multi-input node to an output value;
• representing, during training, each multi-input node in a trainable form parameterized by trainable parameters that are adjustable by an optimization algorithm, the trainable form being configured to compute the node output from the at least two inputs in accordance with current values of the trainable parameters;
• performing a residual initialization by initializing the trainable parameters for each of a plurality of the multi-input nodes such that, prior to iterative training, the corresponding trainable form causes the node output to be determined primarily by a residual operation that propagates information from a single input of the multi-input node to the node output;
• training the node network by iteratively adjusting the trainable parameters using the optimization algorithm based on a training objective;
• wherein, during the training, at least some of the residually initialized multi-input nodes transition such that the corresponding trainable form causes the node output to be determined primarily by a different candidate discrete operation that functionally depends on each of at least two inputs of the multi-input node;
• and
• after the training, generating the discretized form of the node network by applying a discretization rule to the trainable parameters of at least some nodes to select corresponding candidate discrete operations for the at least some nodes.

A computing system for generating a trained node network for performing an inference task, comprising:

• at least one processor;
• and
• a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the computing system to:
• instantiate, in the memory, an untrained node network comprising a plurality of nodes arranged in a network topology, each node configured to receive a plurality of node inputs, including at least a first node input and a second node input, and to produce a node output, wherein each of at least some of the nodes is associated with a respective set of trainable parameters usable to select, for a discretized form of the node network, a node function for the node from a set of candidate node functions;
• define a subset of the plurality of nodes;
• perform residual initialization of the untrained node network by initializing, for each node of the subset of nodes, the trainable parameters such that application of a discretization rule to the trainable parameters selects, for the node, a candidate node function that functionally depends on only a single one of the node inputs;
• iteratively train the node network via a plurality of training iterations, where at least some of the training iterations include updating at least some of the trainable parameters;
• and
• generate, after completion of the plurality of training iterations, a trained node network by applying the discretization rule to the trainable parameters to select, for at least some of the nodes, respective node functions and outputting a fixed representation of the trained node network.

An application-specific integrated circuit (ASIC) with logic circuitry that is an implementation of a fixed logic gate network, manufactured using a process comprising:

• instantiating a node network comprising a plurality of nodes and connections, the plurality of nodes including a set of multi-input nodes each configured to receive at least two inputs and produce a node output;
• for each multi-input node in the set, providing a plurality of candidate discrete operations selectable for use in a discretized form of the node network, each candidate discrete operation being configured to map the at least two inputs of the multi-input node to an output value;
• representing, during training, each multi-input node in a trainable form parameterized by trainable parameters that are adjustable by an optimization algorithm, the trainable form being configured to compute the node output from the at least two inputs in accordance with current values of the trainable parameters;
• performing a residual initialization by initializing the trainable parameters for each of a plurality of the multi-input nodes such that, prior to iterative training, the corresponding trainable form causes the node output to be determined primarily by a residual operation that propagates information from a single input of the multi-input node to the node output;
• training the node network by iteratively adjusting the trainable parameters using the optimization algorithm based on a training objective;
• wherein, during the training, at least some of the residually initialized multi-input nodes transition such that the corresponding trainable form causes the node output to be determined primarily by a different candidate discrete operation that functionally depends on each of at least two inputs of the multi-input node;
• and
• after the training, generating the discretized form of the node network as a fixed logic gate network by applying a discretization rule to the trainable parameters of at least some nodes to select corresponding candidate discrete operations for the at least some nodes.