# US 2024/0005166 A1 · Minimum Deep Learning with Gating Multiplier

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005166 A1
- Document ID: US20240005166A1
- Status: Published application
- Kind: A1
- Application number: 18/467,207
- Filed: 2023-09-14
- Published: 2024-01-04
- Earliest claimed: 2020-03-04
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Gil Shamir
- Web page: https://patentgrep.com/patent/US20240005166A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods according to the present disclosure can employ a computer-implemented method for inference using a machine-learned model. The method can be implemented by a computing system having one or more computing devices. The method can include obtaining data descriptive of a neural network including one or more network units and one or more gating paths, wherein each of the gating path(s) includes one or more gating units. The method can include obtaining data descriptive of one or more input features. The method can include determining one or more network unit outputs from the network unit(s) based at least in part on the input feature(s). The method can include determining one or more gating values from the gating path(s). The method can include determining one or more gated network unit outputs based at least in part on a combination of the network unit output(s) and the gating value(s).

## Claims

### Claim at position 1

1-17. (canceled)

### Claim 18

A computer-implemented method of deploying a machine-learned model, the computer-implemented method comprising:
obtaining, by one or more computing devices, data descriptive of a neural network comprising one or more network units and one or more gating units, the one or more gating units comprising one or more gating paths associated with the one or more network units;
training, by the one or more computing devices, the neural network to learn one or more network parameters of the one or more network units and one or more gating parameters of the one or more gating units;
sparsifying, by the one or more computing devices, the neural network based at least in part on the one or more gating parameters of the one or more gating units to generate a sparsified neural network; and
deploying the sparsified neural network to perform inference.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the one or more gating parameters comprises a gating threshold, wherein sparsifying the neural network comprises dropping one or more network units from the neural network based at least in part on the gating threshold.

### Claim 20 (depends on 18)

The computer-implemented method of claim 18, wherein training the neural network comprises epoch training, wherein a learning rate in subsequent epochs is decremented.

### Claim 21 (depends on 18)

The computer-implemented method claim 18, wherein sparsifying the neural network comprises approximating one or more gating paths based at least in part on the one or more gating parameters.

### Claim 22 (depends on 18)

The computer-implemented method of claim 18, wherein sparsifying the neural network comprises dropping at least one row of a multiplication matrix comprising the neural network.

### Claim 23 (depends on 18)

The computer-implemented method of claim 18, wherein the method further comprises clipping, by the one or more computing devices, one or more outputs of the one or more network units in the sparsified neural network during at least one of a forward propagation step or a backward propagation step after the sparsified neural network is deployed.

### Claim 24 (depends on 18)

The computer-implemented method of claim 18, wherein the one or more gating units comprises at least one of a benefit path configured to produce a benefit score associated with the one or more network units, a scaling function configured to produce the gating value based at least in part on the benefit score, or a clipping function configured to clip the gating value based at least in part on a clipping threshold.

### Claim 25 (depends on 24)

The computer-implemented method of claim 24, wherein the benefit path comprises at least one of a benefit bias unit or a benefit link weight, wherein the one or more gating parameters comprises a benefit score bias, wherein the benefit bias unit is configured to store the benefit score bias, and wherein the benefit path produces the benefit score based at least in part on the benefit score bias, wherein the benefit path comprises at least one of a stateful benefit bias unit or a stateful benefit link weight.

### Claim 26 (depends on 24)

The computer-implemented method of claim 24, wherein the neural network comprises one or more network layers, each of the one or more network layers comprising one or more network units, wherein a first network layer of the one or more network layers comprises the one or more network units, and wherein the benefit path comprises a weighted sum of inputs,
wherein the weighted sum of inputs comprises outputs of the one or more network units.

### Claim 27 (depends on 24)

The computer-implemented method of claim 24, wherein the benefit path comprises one or more benefit path layers, each of the benefit path layers comprising one or more benefit units.

### Claim 28 (depends on 27)

The computer-implemented method of claim 27, wherein at least one of the one or more benefit path layers comprises a bottleneck layer, wherein a dimensionality of the bottleneck layer is less than a dimensionality of a preceding layer of the one or more benefit path layers.

### Claim 29 (depends on 18)

The computer-implemented method of claim 18, wherein the neural network comprises a second gated network unit, wherein a second gated output of the second gated network unit comprises a second intermediate output of the second gated network unit multiplied by the gating value.

### Claim 30 (depends on 18)

The computer-implemented method of claim 18, wherein at least one of the one or more network units and one or more gating units are stateful gating units that are respective to a state of the at least of the one or more network units, wherein the at least one of the one or more network units having the state comprises at least one of an embedding unit, a link weight, or a bias unit.

### Claim 31 (depends on 18)

The computer-implemented method of claim 18, wherein the one or more network units comprises an embedding vector, and wherein the gating value is associated with each embedding component in the embedding vector.

### Claim 32 (depends on 24)

The computer-implemented method of claim 24, wherein the scaling function comprises one of a sigmoid function, a half-sigmoid function, or shifted piecewise smooth activation function.

### Claim 33 (depends on 24)

The computer-implemented method of claim 24, wherein the scaling function comprises a self-gating scaling function.

### Claim 34 (depends on 18)

The computer-implemented method of claim 18, wherein the one or more gating parameters are learned during training of the neural network.

### Claim 35 (depends on 18)

The computer-implemented method of claim 18, wherein the one or more gating units are employed as an activation function for the one or more network units.

### Claim 36

A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining data descriptive of a neural network comprising one or more network units and one or more gating units, the one or more gating units comprising one or more gating paths associated with the one or more network units;
training the neural network to learn one or more network parameters of the one or more network units and one or more gating parameters of the one or more gating units;
sparsifying the neural network based at least in part on the one or more gating parameters of the one or more gating units to generate a sparsified neural network; and
deploying the sparsified neural network to perform inference.

### Claim 37

A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a sparsified neural network that was obtained by:
obtaining data descriptive of a neural network comprising one or more network units and one or more gating units, the one or more gating units comprising one or more gating paths associated with the one or more network units;
training the neural network to learn one or more network parameters of the one or more network units and one or more gating parameters of the one or more gating units; and
sparsifying the neural network based at least in part on the one or more gating parameters of the one or more gating units to generate the sparsified neural network; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving an input;
generating, based on the input, a prediction from the sparsified neural network; and
returning an output based on the prediction.
