# US 2025/0094793 A1 · RE-ARRANGING FEED FORWARD NETWORKS (FFNs) IN TRANSFORMER-BASED MODELS

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094793 A1
- Document ID: US20250094793A1
- Status: Published application
- Kind: A1
- Application number: 18/469,909
- Filed: 2023-09-19
- Published: 2025-03-20
- Earliest claimed: 2023-09-19
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Manish Kumar SINGH; Tianyu JIANG; Hsin-Pai CHENG; Kartikeya BHARDWAJ; Hong CAI; Mingu LEE; Munawar HAYAT; Christopher LOTT; Fatih Murat PORIKLI
- Web page: https://patentgrep.com/patent/US20250094793A1

## CPC

- G06N 3/0499: Feedforward networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A processor-implemented method for image or text processing includes receiving, by an artificial neural network (ANN) model, a set of tokens corresponding to an input. A token interaction block of the ANN model processes the set of tokens according to each channel of the input to generate a spatial mixture of a set of features for each channel of the input. A feed forward network block of the ANN model generates a mixture of channel features based on the spatial mixture of the set of features for each channel of the input. An attention block of the ANN model determines a set of attended features of the mixture of channel features according to a set of attention weights. In turn, the ANN model generates an inference based on the set of attend features of the mixture of channel features.

## Claims

### Claim 1

A processor-implemented method performed by one or more processors, the processor-implemented method comprising:
receiving, by an artificial neural network (ANN) model, a set of tokens corresponding to an input;
processing, by a token interaction block of the ANN model, the set of tokens according to each channel of the input to generate a spatial mixture of a set of features for each channel of the input;
generating, by a feed forward network (FFN) block of the ANN model, a mixture of channel features based on the spatial mixture of the set of features for each channel of the input;
determining, by an attention block of the ANN model, a set of attended features of the mixture of channel features according to a set of attention weights; and
generating, by the ANN model, an inference based on the set of attend features of the mixture of channel features.

### Claim 2 (depends on 1)

The processor-implemented method of claim 1, further comprising processing the set of tokens corresponding to the input using a depth-wise convolution to generate the spatial mixture of the set of features.

### Claim 3 (depends on 1)

The processor-implemented method of claim 1, further comprising processing the spatial mixture of the set of features using a point-wise convolution to generate the mixture of channel features.

### Claim 4 (depends on 1)

The processor-implemented method of claim 1, in which the ANN model comprises a vision transformer, a bi-directional encoder representations from transformers (BERT), a robustly optimized BERT approach (ROBERTa)-based transformer, an XLNet-based transformer, Transformer-XL-based transformer, or a generative pre-trained transformer (GPT).

### Claim 5 (depends on 1)

The processor-implemented method of claim 1, in which the input comprises an image, an audio signal, or a textual input.

### Claim 6 (depends on 5)

The processor-implemented method of claim 5, in which the image is divided into a set of patches, the set of tokens corresponding to the set of patches of the image.

### Claim 7 (depends on 1)

The processor-implemented method of claim 1, in which the inference is a classification of the input.

### Claim 8 (depends on 1)

The processor-implemented method of claim 1, further comprising reshaping the mixture of channel features to form a one-dimensional sequence of mixed features.

### Claim 9

An apparatus, comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
receive, by an artificial neural network (ANN) model, a set of tokens corresponding to an input;
process, by a token interaction block of the ANN model, the set of tokens according to each channel of the input to generate a spatial mixture of a set of features for each channel of the input;
generate, by a feed forward network (FFN) block of the ANN model, a mixture of channel features based on the spatial mixture of the set of features for each channel of the input;
determine, by an attention block of the ANN model, a set of attended features of the mixture of channel features according to a set of attention weights; and
generate, by the ANN model, an inference based on the set of attend features of the mixture of channel features.

### Claim 10 (depends on 9)

The apparatus of claim 9, in which the at least one processor is further configured to process the set of tokens corresponding to the input using a depth-wise convolution to generate the spatial mixture of the set of features.

### Claim 11 (depends on 9)

The apparatus of claim 9, in which the at least one processor is further configured to process the spatial mixture of the set of features using a point-wise convolution to generate the mixture of channel features.

### Claim 12 (depends on 9)

The apparatus of claim 9, in which the ANN model comprises a vision transformer, a bi-directional encoder representations from transformers (BERT), a robustly optimized BERT approach (ROBERTa)-based transformer, an XLNet-based transformer, Transformer-XL-based transformer, or a generative pre-trained transformer (GPT).

### Claim 13 (depends on 9)

The apparatus of claim 9, in which the input comprises an image, an audio signal, or a textual input.

### Claim 14 (depends on 13)

The apparatus of claim 13, in which the at least one processor is further configured to divide the image into a set of patches, the set of tokens corresponding to the set of patches of the image.

### Claim 15 (depends on 9)

The apparatus of claim 9, in which the inference is a classification of the input.

### Claim 16 (depends on 9)

The apparatus of claim 9, in which the at least one processor is further configured to reshape the mixture of channel features to form a one-dimensional sequence of mixed features.

### Claim 17

A non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor and comprising:
program code to receive, by an artificial neural network (ANN) model, a set of tokens corresponding to an input;
program code to process, by a token interaction block of the ANN model, the set of tokens according to each channel of the input to generate a spatial mixture of a set of features for each channel of the input;
program code to generate, by a feed forward network (FFN) block of the ANN model, a mixture of channel features based on the spatial mixture of the set of features for each channel of the input;
program code to determine, by an attention block of the ANN model, a set of attended features of the mixture of channel features according to a set of attention weights; and
program code to generate, by the ANN model, an inference based on the set of attend features of the mixture of channel features.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, in which the program code comprises program code to process the set of tokens corresponding to the input using a depth-wise convolution to generate the spatial mixture of the set of features.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, in which the program code comprises program code to process the spatial mixture of the set of features using a point-wise convolution to generate the mixture of channel features.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, in which the ANN model comprises a vision transformer, a bi-directional encoder representations from transformers (BERT), a robustly optimized BERT approach (ROBERTa)-based transformer, an XLNet-based transformer, Transformer-XL-based transformer, or a generative pre-trained transformer (GPT).

### Claim 21 (depends on 17)

The non-transitory computer-readable medium of claim 17, in which the input comprises an image, an audio signal, or a textual input.

### Claim 22 (depends on 21)

The non-transitory computer-readable medium of claim 21, in which the program code comprises program code to divide the image into a set of patches, the set of tokens corresponding to the set of patches of the image.

### Claim 23 (depends on 17)

The non-transitory computer-readable medium of claim 17, in which the program code comprises program code to reshape the mixture of channel features to form a one-dimensional sequence of mixed features.

### Claim 24

An apparatus comprising:
means for receiving, by an artificial neural network (ANN) model, a set of tokens corresponding to an input;
means for processing, by a token interaction block of the ANN model, the set of tokens according to each channel of the input to generate a spatial mixture of a set of features for each channel of the input;
means for generating, by a feed forward network (FFN) block of the ANN model, a mixture of channel features based on the spatial mixture of the set of features for each channel of the input;
means for determining, by an attention block of the ANN model, a set of attended features of the mixture of channel features according to a set of attention weights; and
means for generating, by the ANN model, an inference based on the set of attend features of the mixture of channel features.

### Claim 25 (depends on 24)

The apparatus of claim 24, further comprising means for processing the set of tokens corresponding to the input using a depth-wise convolution to generate the spatial mixture of the set of features.

### Claim 26 (depends on 24)

The apparatus of claim 24, further comprising means for processing the spatial mixture of the set of features using a point-wise convolution to generate the mixture of channel features.

### Claim 27 (depends on 24)

The apparatus of claim 24, in which the ANN model comprises a vision transformer, a bi-directional encoder representations from transformers (BERT), a robustly optimized BERT approach (ROBERTa)-based transformer, an XLNet-based transformer, Transformer-XL-based transformer, or a generative pre-trained transformer (GPT).

### Claim 28 (depends on 24)

The apparatus of claim 24, in which the input comprises an image, an audio signal, or a textual input.

### Claim 29 (depends on 28)

The apparatus of claim 28, further comprising means for dividing the image into a set of patches, the set of tokens corresponding to the set of patches of the image.

### Claim 30 (depends on 24)

The apparatus of claim 24, further comprising means for reshaping the mixture of channel features to form a one-dimensional sequence of mixed features.
