# US 2026/0099673 A1 · Memory-efficient draft machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0099673 A1
- Document ID: US20260099673A1
- Status: Published application
- Kind: A1
- Application number: 19/051,081
- Filed: 2025-02-11
- Published: 2026-04-09
- Earliest claimed: 2024-10-08
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Mingu LEE; Wonseok JEON; Junyoung PARK; Kanghoon YOON; Christopher LOTT
- Web page: https://patentgrep.com/patent/US20260099673A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/09: Supervised learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed are systems, apparatuses, processes, and computer-readable media for model training. A device may process, using a linear layer, an embedding generated from a first output token and input features to generate first features, wherein the first output token is generated by a previous iteration of a token predictor and wherein the input features are generated by a previous iteration of a decoding layer. A device may process, using the decoding layer, the first features to generate second features having first dimensions. A device may process, using a down-projection layer, the second features to generate third features having second dimensions smaller than the first dimensions. A device may generate, using the token predictor and the third features, a second output token.

## Claims

### Claim 1

An apparatus for using a machine learning model to generate tokens, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
process, using a down-projection layer, input features to generate first features;
process, using a linear layer, the first features, an embedding generated from a first output token, and an additional feature to generate second features, wherein the first output token is generated by a previous iteration of a token predictor and wherein the additional feature is generated by a previous iteration of a decoding layer;
process, using the decoding layer, the second features to generate third features; and
process, using the token predictor, the third features to generate a second output token.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the down-projection layer comprises first parameters, the linear layer comprises second parameters, and the decoding layer comprises third parameters.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the first parameters, the second parameters, and the third parameters are determined based on training the machine learning model, wherein the training comprises:
processing, using the down-projection layer, training input features to generate first training features;
processing, using the linear layer, the first training features and a training embedding generated from a first training token to generate second training features;
processing, using the decoding layer, the second training features to generate third training features;
processing, using an up-projection layer, the third training features to generate fourth training features; and
processing, using the token predictor, the fourth training features to generate a training output token.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the at least one processor is configured to process, using the down-projection layer and a stop gradient, a target feature to generate an updated regression loss.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein parameters of the token predictor are generated based on merging the up-projection layer with the token predictor.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the input features are generated from a previous iteration of the decoding layer.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the machine learning model is a large language model (LLM).

### Claim 8

A method of using a machine learning model to generate tokens, the method comprising:
processing, using a down-projection layer, input features to generate first features;
processing, using a linear layer, the first features, an embedding generated from a first output token, and an additional feature to generate second features, wherein the first output token is generated by a previous iteration of a token predictor and wherein the additional feature is generated by a previous iteration of a decoding layer;
processing, using the decoding layer, the second features to generate third features; and
processing, using the token predictor, the third features to generate a second output token.

### Claim 9 (depends on 8)

The method of claim 8, wherein the down-projection layer comprises first parameters, the linear layer comprises second parameters, and the decoding layer comprises third parameters.

### Claim 10 (depends on 9)

The method of claim 9, wherein the first parameters, the second parameters, and the third parameters are determined based on training the machine learning model, wherein the training comprises:
processing, using the down-projection layer, training input features to generate first training features;
processing, using the linear layer, the first training features and a training embedding generated from a first training token to generate second training features;
processing, using the decoding layer, the second training features to generate third training features;
processing, using an up-projection layer, the third training features to generate fourth training features; and
processing, using the token predictor, the fourth training features to generate a training output token.

### Claim 11 (depends on 10)

The method of claim 10, further comprising processing, using the down-projection layer and a stop gradient, a target feature to generate an updated regression loss.

### Claim 12 (depends on 10)

The method of claim 10, wherein parameters of the token predictor are generated based on merging the up-projection layer with the token predictor.

### Claim 13 (depends on 8)

The method of claim 8, wherein the input features are generated from a previous iteration of the decoding layer.

### Claim 14 (depends on 8)

The method of claim 8, wherein the machine learning model is a large language model (LLM).

### Claim 15

A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:
process, using a down-projection layer of a machine learning model, input features to generate first features;
process, using a linear layer, the first features, an embedding generated from a first output token, and an additional feature to generate second features, wherein the first output token is generated by a previous iteration of a token predictor and wherein the additional feature is generated by a previous iteration of a decoding layer;
process, using the decoding layer, the second features to generate third features; and
process, using the token predictor, the third features to generate a second output token.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the down-projection layer comprises first parameters, the linear layer comprises second parameters, and the decoding layer comprises third parameters.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the first parameters, the second parameters, and the third parameters are determined based on training the machine learning model, wherein the training comprises:
processing, using the down-projection layer, training input features to generate first training features;
processing, using the linear layer, the first training features and a training embedding generated from a first training token to generate second training features;
processing, using the decoding layer, the second training features to generate third training features;
processing, using an up-projection layer, the third training features to generate fourth training features; and
processing, using the token predictor, the fourth training features to generate a training output token.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by the at least one processor, cause the at least one processor to process, using the down-projection layer and a stop gradient, a target feature to generate an updated regression loss.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein parameters of the token predictor are generated based on merging the up-projection layer with the token predictor.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the input features are generated from a previous iteration of the decoding layer.
