US 2026/0310597 A1 · DISTRIBUTED MEMORY WITH SUBSETS OF MODEL WEIGHTS OR PARAMETERS BASED ON INFERENCE TASK TYPES

Abstract

Technologies for distributed memory with subsets of model weights or parameters based on inference task type are described. One programmable network device receives, from a first inference node over a host interface, a first indication of a first inference task being executed by the first inference node. The programmable network device determines, using the first indication of the first inference task, a first inference task type for the first inference task, and fetches, from the one or more remote nodes over the network, only a first subset of model weights or parameters of an inference model, the first subset of model weights or parameters being associated with the first inference task type. The programmable network device loads the first subset of model weights or parameters into local memory of the first inference node.

Claims20

Dependent claims

A programmable network device comprising:

a processing unit;
memory operatively coupled to the processing unit;
a host interface operatively coupled to the processing unit and a first inference node;
and
a network interface operatively coupled to the processing unit and one or more remote nodes over a network, wherein the memory is to store instructions that, when executed by the processing unit, cause the programmable network device to:
receive, from the first inference node over the host interface, a first indication of a first inference task being executed by the first inference node;
determine, using the first indication of the first inference task, a first inference task type for the first inference task;
fetch, from the one or more remote nodes over the network, only a first subset of model weights or parameters of an inference model, the first subset of model weights or parameters being associated with the first inference task type;
and
load the first subset of model weights or parameters into local memory of the first inference node.

7 more elements

A programmable network device comprising distributed memory logic to:

receive, from a first inference node over a host interface, a first indication of a first inference task being executed by the first inference node;
determine using the first indication of the first inference task, a first inference task type for the first inference task;
fetch, from the one or more remote nodes over a network interface, only a first subset of model weights or parameters of an inference model, the first subset of model weights or parameters being associated with the first inference task type;
and
load the first subset of model weights or parameters into local memory of the first inference node.

2 more elements

A method comprising:

receiving, from a first inference node over a host interface, a first indication of a first inference task being executed by the first inference node;
determining using the first indication of the first inference task, a first inference task type for the first inference task;
fetching, from the one or more remote nodes over a network interface, only a first subset of model weights or parameters of an inference model, the first subset of model weights or parameters being associated with the first inference task type;
and
loading the first subset of model weights or parameters into local memory of the first inference node.

2 more elements

Dependencies are inferred from the claim text.

Distributed memory with subsets of model weights or parameters based on inference task types

Life of this patent

Distributed memory with subsets of model weights or parameters based on inference task types

Life of this patent

Abstract

Claims20