Back to results
US 2026/0310348 A1Published application
Distributed memory for inference operations
AssigneeMellanox Technologies, Ltd.
InventorsAlireza Farshin, Omri Kahalon, Liran Liss, Aviad Shaul Yehezkel, Vishwanath Venkatesan
Life of this patent
Estimating the term…
Abstract
Technologies for distributed memory for inference operations are described. One programmable network device includes a distributed hierarchical memory manager that handles operations related to memory and coherency between an inference node and one or more remote nodes.
Claims20
A programmable network device comprising:
- a processing unit;
- memory operatively coupled to the processing unit;
- a host interface operatively coupled to the processing unit and an inference node;
- and
- a network interface operatively coupled to the processing unit and one or more remote nodes over a network, wherein the memory is to store instructions that, when executed by the processing unit, cause the programmable network device to operate as a distributed hierarchical memory manager to handle operations related to memory and coherency between the inference node and the one or more remote nodes.
2 more elements
A programmable network device comprising distributed memory logic to:
- receive configuration data identifying a plurality of memory nodes, each memory node located at an inference node or at one or more remote nodes;
- determine, using the configuration data, a distributed memory hierarchy that organizes the plurality of memory nodes into a plurality of memory levels for the inference node;
- configure each of the plurality of memory nodes according to the distributed memory hierarchy;
- receive one or more requests from the inference node over a host interface;
- and
- interact with one or more of the plurality of memory nodes over a network interface to perform memory operations for the inference node.
3 more elements
A system for performing inference operations, the system comprising:
- an inference node comprising an inference processor with local memory, additional memory, and zero or more inference processors;
- a remote node comprising remote memory;
- and
- a programmable device coupled to the inference node and the remote node, wherein the programmable device is to:
- receive configuration data identifying a plurality of memory nodes, each memory node located at the inference node or the remote node, wherein the plurality of memory nodes comprises the local memory, the additional memory, and the remote memory;
- determine, using the configuration data, a distributed memory hierarchy that organizes the plurality of memory nodes into a plurality of memory levels for the inference processor;
- configure each of the plurality of memory nodes according to the distributed memory hierarchy;
- receive one or more requests from the inference processor over a host interface;
- and
- interact with one or more of the plurality of memory nodes over a network interface to perform memory operations for the inference processor.
7 more elements
A method of operating a programmable network device, the method comprising:
- receiving configuration data identifying a plurality of memory nodes, each memory node located at an inference node or at one or more remote nodes, each node being operatively coupled to the programmable network device;
- determining, using the configuration data, a distributed memory hierarchy that organizes the plurality of memory nodes into a plurality of memory levels for the inference node;
- configuring each of the plurality of memory nodes according to the distributed memory hierarchy;
- receiving one or more requests from the inference node;
- and
- interacting with one or more of the plurality of memory nodes to perform memory operations for the inference node.
3 more elements
Dependencies are inferred from the claim text.