Inference Context Data Staging Across UALink and CXL Protocol Domains

A method for staging inference context data across protocol domain boundaries, comprising:

• reading, by a host, inference context data from a Compute Express Link (CXL) memory device via CXL.mem;
• sending, by the host, a CXL.mem Master-to-Subordinate (M2S) request to a resource provisioning unit (RPU), the CXL.mem M2S request associated with the inference context data and comprising a first physical address;
• translating, by the RPU, the CXL.mem M2S request to an Ultra Accelerator Link (UALink) Protocol Level Interface (UPLI) request comprising a second physical address;
• and
• sending, by the RPU, the UPLI request to an accelerator via a UALink port;
• wherein the inference context data is staged from the CXL memory device to a local memory of the accelerator across a CXL protocol domain and a UALink protocol domain.

A system comprising:

• an Ultra Accelerator Link (UALink) switch;
• accelerators coupled to the UALink switch, each accelerator of the accelerators comprising a local memory;
• a Compute Express Link (CXL) memory device;
• a resource provisioning unit (RPU) coupled to the UALink switch and to the CXL memory device, the RPU configured to translate between UALink Protocol Level Interface (UPLI) and CXL.mem;
• and
• a host coupled to the CXL memory device via CXL.mem;
• wherein the system is configured to migrate inference context data between the local memory of at least one accelerator of the accelerators and the CXL memory device across a UALink protocol domain and a CXL protocol domain.

A method for migrating inference context data across protocol domain boundaries, comprising:

• sending, by an accelerator, a request conforming to an NVLink-based protocol to a resource provisioning unit (RPU), the request associated with the inference context data stored in a local memory of the accelerator, the request comprising a first physical address;
• translating, by the RPU, the request to a Compute Express Link (CXL) CXL.mem Master-to-Subordinate (M2S) request comprising a second physical address;
• and
• sending, by the RPU, the CXL.mem M2S request to a CXL memory device;
• wherein the inference context data is migrated between the local memory of the accelerator and the CXL memory device across an NVLink protocol domain and a CXL protocol domain.