Back to results
US 2026/0278352 A1Published application
Student-Teacher Distillation for Training and Deploying Differentiable Logic Gate Networks
AssigneeDiffLogic Inc.
InventorFelix Petersen
Life of this patent
Estimating the term…
Abstract
Systems and methods are described for training a logic gate network using student-teacher learning. In various embodiments, a system obtains a teacher model that produces a target output distribution for an input and records a training dataset of inputs and teacher outputs. The system instantiates a differentiable logic-gate network with nodes parameterized over candidate logic gate operators. The system trains the differentiable logic gate network by minimizing the divergence between its outputs and the teacher outputs. After training, a logic gate operator may be selected for each node. The logic gate network may be implemented as logic circuitry in a programmable device or as a fixed-in-silicon application-specific integrated circuit.
Claims24
A computing system, comprising:
- at least one processor;
- and
- a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the computing system to:
- receive a training data set comprising inputs;
- obtain a teacher model configured to generate, for each input, corresponding teacher output values comprising teacher predictions;
- instantiate, in the memory, an untrained student logic gate network with a plurality of nodes, wherein each node is parameterized by a set of differentiable parameters associated with a predefined set of potential logic gate operators;
- iteratively train the student logic gate network via a plurality of training iterations, each training iteration including:
- determining, using the teacher model, teacher output values for a batch of the inputs;
- forward-propagating the batch of the inputs through the student logic gate network to generate a training network output by, for each node, computing an output activation that is a function of input activations of the respective node according to current differentiable parameters thereof;
- computing a loss value that quantifies a difference between the training network output and the teacher output values for the batch;
- determining, via a gradient-based training optimization algorithm, updated differentiable parameters for at least one node;
- and
- applying the updated differentiable parameters to the at least one node;
- and
- generating, after completion of the plurality of training iterations, a fixed student logic gate network by selecting a single logic gate operator from the set of potential logic gate operators for a plurality of the nodes based on the differentiable parameters thereof.
12 more elements
The system of claim Error! Reference source not found., wherein at least one processor is configured to, after training the teacher model, execute the teacher model in an environment to generate (i) a set of input observations comprising at least a portion of the inputs and (ii) corresponding teacher output values, and utilize the set of input observations and the corresponding teacher output values as a distillation data set.
A method for training a differentiable logic gate network, comprising:
- receiving, at a computing system, a training data set comprising input vectors for an inference task;
- obtaining a teacher model configured to generate, for each input vector, corresponding teacher output values comprising teacher predictions for the inference task;
- instantiating, in a memory of the computing system, an untrained student logic gate network with a plurality of nodes, wherein each node is parameterized by a set of differentiable parameters corresponding to a predefined set of potential logic gate operators;
- iteratively training the student logic gate network via a plurality of training iterations, each training iteration including:
- determining teacher output values for a batch of the input vectors by evaluating the teacher model using the batch of the input vectors;
- forward-propagating the batch of the input vectors through the student logic gate network to generate a training network output by, for each node, computing a differentiable output that is a function of the outputs of the potential logic gate operators of the respective node according to current differentiable parameters thereof;
- computing a loss value that quantifies a difference between the training network output and the corresponding teacher output values for the batch;
- determining, via a training optimization algorithm, updated differentiable parameters for at least one node;
- and
- applying the updated differentiable parameters to the at least one node;
- and
- generating, after completion of the plurality of training iterations, a fixed student logic gate network by identifying a single logic gate from the set of potential logic gate operators for a plurality of the nodes based on the differentiable parameters thereof.
9 more elements
A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations comprising:
- receiving a training data set comprising input vectors for an inference task;
- obtaining a teacher model configured to generate, for each input vector, corresponding teacher output values comprising teacher predictions for the inference task;
- instantiating an untrained student logic gate network with a plurality of nodes, wherein each node is parameterized by a set of differentiable parameters corresponding to a predefined set of potential logic gate operators;
- iteratively training the student logic gate network via a plurality of training iterations, each training iteration including:
- determining teacher output values for a batch of the input vectors by evaluating the teacher model using the batch of the input vectors;
- forward-propagating the batch of the input vectors through the student logic gate network to generate a training network output by, for each node, computing a differentiable output that is a function of outputs of the potential logic gate operators of the respective node according to current differentiable parameters thereof;
- computing a loss value that quantifies a difference between the training network output and the corresponding teacher output values for the batch;
- determining, via a training optimization algorithm, updated differentiable parameters for at least one node;
- and
- applying the updated differentiable parameters to the at least one node;
- and
- generating, after completion of the plurality of training iterations, a fixed student logic gate network by identifying a single logic gate from the set of potential logic gate operators for a plurality of the nodes based on the differentiable parameters thereof.
9 more elements
An application-specific integrated circuit (ASIC) with logic circuitry that is an implementation of a fixed logic gate network, manufactured using a process comprising:
- receiving, at a computing system, a training data set comprising input vectors for an inference task;
- obtaining a teacher model configured to generate, for each input vector, corresponding teacher output values comprising teacher predictions for the inference task;
- instantiating, in a memory of the computing system, an untrained student logic gate network with a plurality of nodes, wherein each node is parameterized by a set of differentiable parameters corresponding to a predefined set of potential logic gate operators;
- iteratively training the student logic gate network via a plurality of training iterations, each training iteration including:
- determining teacher output values for a batch of the input vectors by evaluating the teacher model using the batch of the input vectors;
- forward-propagating the batch of the input vectors through the student logic gate network to generate a training network output by, for each node, computing a differentiable output that is a function of the outputs of the potential logic gate operators of the respective node according to current differentiable parameters thereof;
- computing a loss value that quantifies a difference between the training network output and the corresponding teacher output values for the batch;
- determining, via a training optimization algorithm, updated differentiable parameters for at least one node;
- and
- applying the updated differentiable parameters to the at least one node;
- and
- generating, after completion of the plurality of training iterations, a fixed logic gate network by identifying a single logic gate from the set of potential logic gate operators for a plurality of the nodes based on the differentiable parameters thereof.
9 more elements
Dependencies are inferred from the claim text.