US 2026/0270324 A1 · SYSTEMS AND METHODS FOR TOKEN CACHING IN DISTRIBUTED NETWORKS

Abstract

Distributed caching of token representations of artificial intelligence inquiries and corresponding responses across a network core cache and network edge caches is provided. A request may be received from user equipment. A network edge cache at a network edge compute node is evaluated to determine whether any edge cache entries are semantically similar to a vector representation of tokens within the request. If an edge cache entry is identified as being semantically similar, then a cached response of the edge cache entry is provided back to the user equipment. Otherwise, a network core cache is evaluated to see if a core cache entry is semantically similar. If a core cache entry is identified as being semantically similar, then a cached response of the core cache entry is provided back to the user equipment and is cached at the network edge cache. Otherwise, compute functionality is invoked to process the request.

Claims20

Dependent claims

A method, comprising:

receiving a request from user equipment that includes tokens for processing by compute functionality hosted by a compute environment;
evaluating a network edge cache hosted at a network edge compute node of a network to determine whether edge cache entries of cached vector representations of tokens and cached responses generated for the tokens are semantically similar to a vector representation of the tokens;
in response to identifying an edge cache entry having a semantic similarity to the vector representation of the tokens that exceeds a threshold, providing the user equipment with a cached response of the edge cache entry;
in response to not identifying at least one edge cache entry with a semantic similarity that exceeds the threshold, routing the request to a network core cache of the network;
and
in response to identifying a core cache entry having a semantic similarity to the vector representation of the tokens that exceeds the threshold, providing a cached response of the core cache entry to the network edge cache to store as a new edge cache entry and to provide to the user equipment.

3 more elements

A system, comprising:

one or more processors configured for executing instructions to perform operations comprising:
receiving a request from user equipment that includes a tokens targeting compute functionality hosted by a compute environment;
evaluating a network edge cache hosted at a network edge compute node of a network to determine whether edge cache entries of cached vector representations of tokens and cached responses generated for the tokens are semantically similar to a vector representation of the tokens;
in response to identifying an edge cache entry having a semantic similarity to the vector representation of the tokens that exceeds a threshold, providing the user equipment with a cached response of the edge cache entry;
in response to not identifying at least one edge cache entry with a semantic similarity that exceeds the threshold, routing the request to a network core cache of the network;
and
in response to identifying a core cache entry having a semantic similarity to the vector representation of the tokens that exceeds the threshold, providing a cached response of the core cache entry to the network edge cache to store as a new edge cache entry and to provide to the user equipment.

4 more elements

A non-transitory computer-readable medium storing instructions that when executed by one or more processors facilitate performance of operations comprising:

receiving a request from user equipment that includes a tokens targeting compute functionality hosted by a compute environment;
evaluating a network edge cache hosted at a network edge compute node of a network to determine whether edge cache entries of cached vector representations of tokens and cached responses generated for the tokens are semantically similar to a vector representation of the tokens;
in response to identifying an edge cache entry having a semantic similarity to the vector representation of the tokens that exceeds a threshold, providing the user equipment with a cached response of the edge cache entry;
in response to not identifying at least one edge cache entry with a semantic similarity that exceeds the threshold, routing the request to a network core cache of the network;
and
in response to identifying a core cache entry having a semantic similarity to the vector representation of the tokens that exceeds the threshold, providing a cached response of the core cache entry to the network edge cache to store as a new edge cache entry and to provide to the user equipment.

3 more elements

Dependencies are inferred from the claim text.

Systems and methods for token caching in distributed networks

Life of this patent

Systems and methods for token caching in distributed networks

Life of this patent

Abstract

Claims20