# US 2025/0053815 A1 · Neural networks with switch layers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0053815 A1
- Document ID: US20250053815A1
- Status: Published application
- Kind: A1
- Application number: 18/806,647
- Filed: 2024-08-15
- Published: 2025-02-13
- Earliest claimed: 2021-01-07
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: William Bradley Fedus; Barret Zoph; Noam M. Shazeer
- Web page: https://patentgrep.com/patent/US20250053815A1

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network including one or more switch layers.

## Claims

### Claim 1

(canceled)

### Claim 2

A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
an attention neural network configured to perform a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions; and
generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions,
wherein, for each of one or more of the plurality of layers, the feed-forward layer of the layer includes a respective switch layer that comprises:
(i) a respective learned routing function having routing parameters, and
(ii) a respective plurality of expert neural networks each having a respective set of expert parameters,
wherein the respective switch layer is configured to:
for each attended layer input in the attended input sequence generated by the attention sub-layer of the layer:
apply the respective learned routing function in the switch layer to the attended layer input in accordance with current values of the routing parameters of the respective learned routing function to generate a score distribution that includes a respective routing score for each of the plurality of expert neural networks in the switch layer;
select, from the plurality of expert neural networks, only one of the expert neural networks using the score distribution;
process the attended layer input using only the selected expert neural network and in accordance with current values of the expert parameters of the selected expert neural network to generate an initial switch layer output for the attended layer input; and
generate a final switch layer output for the switch layer for the attended layer input from the initial switch layer output generated by the selected expert neural network.

### Claim 3 (depends on 2)

The system of claim 2, wherein generating a final switch layer output for the switch layer for the attended layer input from the initial switch layer output generated by the selected expert neural network comprises computing a product of the routing score for the selected expert neural network and the initial switch layer output generated by the selected expert neural network.

### Claim 4 (depends on 2)

The system of claim 2, wherein selecting, from the plurality of expert neural networks, only one of the expert neural networks using the score distribution comprises:
selecting only an expert neural network having a highest routing score.

### Claim 5 (depends on 2)

The system of claim 2, where one or more of the plurality of layers includes a feed-forward sub-layer that does not comprise a switch layer.

### Claim 6 (depends on 4)

The system of claim 4, wherein the feed-forward layer that includes the respective switch layer is configured to:
generate an output for the layer from the respective layer outputs from the final switch layer outputs generated by the respective switch layer for the attended layer inputs.

### Claim 7 (depends on 6)

The system of claim 6, wherein generating the output for the layer comprises:
applying a residual connection and normalization to the final switch layer outputs to generate the output.

### Claim 8 (depends on 3)

The system of claim 3, wherein the system includes a plurality of hardware devices, and wherein implementing the neural network comprises:
for each switch layer, executing each expert neural network in the switch layer on a different one of the plurality of hardware devices.

### Claim 9 (depends on 2)

The system of claim 2, wherein, for each switch layer, the instructions cause the one or more computers to perform at least some of the operations of the routing function using a first computer number format that occupies a first number of bits in memory and perform the operations of the expert neural networks using a second computer number format that occupies a second, smaller number of bits in memory.

### Claim 10 (depends on 9)

The system of claim 9, wherein the first computer number format is float32 and the second computer number format is bfloat16.

### Claim 11 (depends on 2)

The system of claim 2, wherein, for each switch layer, the respective learned routing function in the switch layer is configured to:
apply a matrix of the routing parameters of the respective learned routing function to the attended layer input to generate a respective logit for each of the plurality of expert neural networks in the switch layer; and
apply a softmax to an input that comprises (i) the respective logits or (ii) noisy logits generated by applying noise to the respective logits to generate the score distribution.

### Claim 12 (depends on 11)

The system of claim 11, wherein the switch layer is configured to convert the input to the softmax from a second computer number format to a first computer number format and convert the routing score for the selected expert neural network to the second computer number format prior to computing the product of the routing score and the initial switch layer output.

### Claim 13

A method performed by one or more computers, the method comprising:
receiving a network input; and
processing the network input using an attention neural network configured to perform a machine learning task to generate a network output for the network input, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions; and
generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions,
wherein, for each of one or more of the plurality of layers, the feed-forward layer of the layer includes a respective switch layer that comprises:
(i) a respective learned routing function having routing parameters, and
(ii) a respective plurality of expert neural networks each having a respective set of expert parameters,
wherein the respective switch layer is configured to:
for each attended layer input in the attended input sequence generated by the attention sub-layer of the layer:
apply the respective learned routing function in the switch layer to the attended layer input in accordance with current values of the routing parameters of the respective learned routing function to generate a score distribution that includes a respective routing score for each of the plurality of expert neural networks in the switch layer;
select, from the plurality of expert neural networks, only one of the expert neural networks using the score distribution;
process the attended layer input using only the selected expert neural network and in accordance with current values of the expert parameters of the selected expert neural network to generate an initial switch layer output for the attended layer input; and
generate a final switch layer output for the switch layer for the attended layer input from the initial switch layer output generated by the selected expert neural network.

### Claim 14 (depends on 13)

The method of claim 13, wherein generating a final switch layer output for the switch layer for the attended layer input from the initial switch layer output generated by the selected expert neural network comprises computing a product of the routing score for the selected expert neural network and the initial switch layer output generated by the selected expert neural network.

### Claim 15 (depends on 13)

The method of claim 13, wherein selecting, from the plurality of expert neural networks, only one of the expert neural networks using the score distribution comprises:
selecting only an expert neural network having a highest routing score.

### Claim 16 (depends on 13)

The method of claim 13, where one or more of the plurality of layers includes a feed-forward sub-layer that does not comprise a switch layer.

### Claim 17 (depends on 16)

The method of claim 16, wherein the feed-forward layer that includes the respective switch layer is configured to:
generate an output for the layer from the respective layer outputs from the final switch layer outputs generated by the respective switch layer for the attended layer inputs.

### Claim 18 (depends on 17)

The method of claim 17, wherein generating the output for the layer comprises:
applying a residual connection and normalization to the final switch layer outputs to generate the output.

### Claim 19 (depends on 13)

The method of claim 13, wherein the system includes a plurality of hardware devices, and wherein implementing the neural network comprises:
for each switch layer, executing each expert neural network in the switch layer on a different one of the plurality of hardware devices.

### Claim 20 (depends on 13)

The method of claim 13, wherein, for each switch layer, the instructions cause the one or more computers to perform at least some of the operations of the routing function using a first computer number format that occupies a first number of bits in memory and perform the operations of the expert neural networks using a second computer number format that occupies a second, smaller number of bits in memory.

### Claim 21

One or more non-transitory computer-readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving a network input; and
processing the network input using an attention neural network configured to perform a machine learning task to generate a network output for the network input, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions; and
generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions,
wherein, for each of one or more of the plurality of layers, the feed-forward layer of the layer includes a respective switch layer that comprises:
(i) a respective learned routing function having routing parameters, and
(ii) a respective plurality of expert neural networks each having a respective set of expert parameters,
wherein the respective switch layer is configured to:
for each attended layer input in the attended input sequence generated by the attention sub-layer of the layer:
apply the respective learned routing function in the switch layer to the attended layer input in accordance with current values of the routing parameters of the respective learned routing function to generate a score distribution that includes a respective routing score for each of the plurality of expert neural networks in the switch layer;
select, from the plurality of expert neural networks, only one of the expert neural networks using the score distribution;
process the attended layer input using only the selected expert neural network and in accordance with current values of the expert parameters of the selected expert neural network to generate an initial switch layer output for the attended layer input; and
generate a final switch layer output for the switch layer for the attended layer input from the initial switch layer output generated by the selected expert neural network.
