Heterogeneous relaxations in differentiable logic gate networks

An application-specific integrated circuit (ASIC), comprising:

• logic circuitry implementing a fixed logic gate network for performing an inference task, the fixed logic gate network comprising a plurality of nodes, including:
• a first set of nodes, each node of the first set implementing a respective selected logic gate operator selected from a respective predefined set of potential logic gate operators;
• and
• a second set of nodes, each node of the second set implementing a respective predefined logic gate operator, wherein the second set of nodes comprises a plurality of repeating structural inductive bias gates arranged at multiple locations in the fixed logic gate network to impose structural inductive bias on the fixed logic gate network;
• wherein the fixed logic gate network implemented by the logic circuitry was generated by a training process comprising:
• instantiating an untrained node network comprising a plurality of nodes, wherein:
• (i) each node of a first set of nodes of the plurality of nodes is parameterized by a set of differentiable parameters associated with a respective predefined set of potential logic gate operators, and
• (ii) each node of a second set of nodes of the plurality of nodes corresponds to a respective predefined logic gate operator;
• iteratively training the node network over a plurality of training iterations, each training iteration including:
• computing, for each node in the first set of nodes, a respective node output based on the differentiable parameters thereof using a first relaxation for selecting among the set of potential logic gate operators,
• computing, for each node in the second set of nodes, a respective node output based on the respective predefined logic gate operator using a second relaxation different from the first relaxation, and
• applying updated differentiable parameters to at least one node in the first set of nodes;
• and
• generating, after completion of the plurality of training iterations, the fixed logic gate network based on (i) the predefined logic gate operators of the second set of nodes and (ii) selection of a single logic gate operator from the set of potential logic gate operators based on the differentiable parameters for at least some nodes of the first set of nodes.