# US 2026/0037841 A1 · Systems and methods for processing requests for a machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0037841 A1
- Document ID: US20260037841A1
- Status: Published application
- Kind: A1
- Application number: 19/224,245
- Filed: 2025-05-30
- Published: 2026-02-05
- Earliest claimed: 2024-08-05
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Yu Gong; Andrew Chang; Hingkwan Huen
- Web page: https://patentgrep.com/patent/US20260037841A1

## CPC

- G06N 3/063: using electronic means
- G06N 5/04: Inference or reasoning models
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system comprising: a processing circuit; and a memory storing instructions, which, based on being executed by the processing circuit, cause the processing circuit to perform: identifying a first computation performed by a machine learning model; scheduling a first memory access task associated with a first portion of a first data with respect to the first computation; identifying a second computation performed by the machine learning model, wherein the second computation and the first computation are separate computations; and scheduling a second memory access task associated with a second portion of the first data with respect to the second computation, wherein the first portion and the second portion are different portions of the first data.

## Claims

### Claim 1

A method comprising:
receiving one or more input tokens associated with a one or more first requests to a machine learning model;
receiving one or more output tokens associated with one or more second requests to the machine learning model;
associating a first portion of the one or more input tokens and a first portion of the one or more output tokens with a first group;
associating a second portion of the one or more input tokens and a second portion of the one or more output tokens with a second group; and
processing, by the machine learning model, the first group and the second group for generating an inference.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more input tokens comprise a set of tokens associated with a first request of the one or more first requests and a set of tokens associated with a second request of the one or more first requests, and wherein the first portion of the one or more input tokens comprises the set of tokens associated with the first request.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first portion of the one or more input tokens further comprises the set of tokens associated with the second request.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more input tokens comprises a first set of tokens associated with a first request of the one or more first requests and a second set of tokens associated with a second request of the one or more first requests, and wherein the first portion of the one or more input tokens includes a first portion of the first set of tokens and the second portion of the one or more input tokens includes a second portion of the first set of tokens.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first portion of the one or more input tokens includes a first portion of the second set of tokens and the second portion of the one or more input tokens includes a second portion of the second set of tokens.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first portion of the one or more output tokens includes a first set of tokens associated with a first request of the one or more second requests, and the second portion of the one or more output tokens includes a second set of tokens associated with a second request of the one or more second requests.

### Claim 7 (depends on 1)

The method of claim 1, wherein the one or more first requests include one or more first input queries and the one or more second requests include one or more second input queries.

### Claim 8 (depends on 7)

The method of claim 7, wherein the one or more input tokens are generated based on processing the one or more first input queries, and the output tokens are generated based on executing a neural network to make a prediction based on the one or more second input queries.

### Claim 9

A method comprising:
identifying a first computation performed by a machine learning model;
scheduling a first memory access task associated with a first portion of a first data with respect to the first computation;
identifying a second computation performed by the machine learning model, wherein the second computation and the first computation are separate computations; and
scheduling a second memory access task associated with a second portion of the first data with respect to the second computation, wherein the first portion and the second portion are different portions of the first data.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
performing the first computation and the first memory access task according to a schedule; and
performing the second computation and the second memory access task according to the schedule.

### Claim 11 (depends on 9)

The method of claim 9, wherein first computation is associated with a first group of tokens and a first layer of the machine learning model, wherein the second computation is associated with a second group of tokens and the first layer.

### Claim 12 (depends on 11)

The method of claim 11, wherein the first data includes layer weight data associated with a second layer of the machine learning model.

### Claim 13 (depends on 12)

The method of claim 12, wherein one or more computations of the machine learning model, including the first computation and the second computation, are associated with K number of groups of tokens, the method comprising:
separating the layer weights data into K portions; and
scheduling the K portions of the layer weights data with respect to the K groups of tokens.

### Claim 14 (depends on 9)

The method of claim 9, wherein first computation is associated with a first group of tokens and a first layer of the machine learning model, wherein the second computation is associated with a first group of tokens and a second layer of the machine learning model.

### Claim 15 (depends on 14)

The method of claim 14, wherein the first data includes key-value data associated with a third group of tokens.

### Claim 16 (depends on 15)

The method of claim 15, wherein one or more computations of the machine learning model, including the first computation and the second computation, are associated with M number of layers of the machine learning model, the method comprising:
separating the key-value data into M portions; and
scheduling memory access tasks associated with the M portions with respect to the M layers.

### Claim 17 (depends on 14)

The method of claim 14, wherein the first layer and the second layer are layers of a transformer layer of a large language model.

### Claim 18 (depends on 14)

The method of claim 14, wherein the first layer is a self-attention layer of a large language model of the machine learning model, and the second layer is a feed forward neural-network layer of the large language model.

### Claim 19

A system comprising:
a processing circuit; and
a memory storing instructions, which, based on being executed by the processing circuit, cause the processing circuit to perform:
identifying a first computation performed by a machine learning model;
scheduling a first memory access task associated with a first portion of a first data with respect to the first computation;
identifying a second computation performed by the machine learning model, wherein the second computation and the first computation are separate computations; and
scheduling a second memory access task associated with a second portion of the first data with respect to the second computation, wherein the first portion and the second portion are different portions of the first data.

### Claim 20 (depends on 19)

The system of claim 19, wherein the instructions, based on being executed by the processing circuit, further cause the processing circuit to perform:
performing the first computation and the first memory access task according to a schedule; and
performing the second computation and the second memory access task according to the schedule.
