US 2026/0087329 A1 · PERSISTENT FIXED-SIZE MEMORY FOR MACHINE-LEARNING USING RECURRENT NEURAL NETWORKS

Abstract

Computer-implemented methods and systems provide a persistent fixed-size recurrent memory that supports long or unbounded sequence processing with substantially constant compute and bounded memory. A recurrent model maintains a matrix state updated per chunk from key, value, gate, and optional control signals; a gated error between value and a key-weighted state yields a rank-1 proposal. Chunk proposals are reconciled coordinate-wise by an order-invariant convex combiner; solitary proposals pass unchanged. Queries can be answered from proposals without materializing a full updated state. Multiple asynchronous agents emit sparse updates with overwrite strengths that are merged using sparse convex or max selection and optional optimizer preprocessing. A feedback projection writes higher layer activations into lower layer state to consolidate durable skills and reduce forgetting. The approach enables durable retention, parallel inference-time adaptation, scalable linear-attention approximation, and persistent fixed memory use over arbitrarily long sequences.

Claims28

Dependent claims

A computer-implemented method for approximating a linear attention mechanism in a chunkwise manner, the method comprising:

maintaining, in memory, a matrix-valued state;
receiving a chunk comprising, for each input, a key vector, a value vector, a gate parameter that is a scalar or a vector, and optionally an independent per-input control value, and further including a set of query vectors;
for each input in the chunk:
computing an error as a difference between the value vector and a product of the state and the key vector;
multiplicatively applying the gate parameter to the error, element-wise when the gate parameter is vector-valued, to obtain a gated error;
and forming an update proposal as a rank-1 outer product of the gated error and the key vector;
reconciling, at each coordinate of the state, the update proposals into a chunk update by computing per-coordinate non-negative coefficients over the set of update proposals such that (i) when exactly one proposal contributes at that coordinate the coefficient for that proposal equals one and the others equal zero, and (ii) when two or more proposals contribute at that coordinate the coefficients sum to one, thereby yielding a value within the convex hull of those proposals;
and updating the state with the chunk update;
and for each query vector, producing an output vector using the state.

6 more elements

A computer-implemented method comprising:

(a) maintaining, in memory, a state tensor for a recurrent neural network (the “state”);
(b) initializing a plurality of agents with a common initial state and running each agent along a trajectory that is a sequence of inputs and timesteps beginning at the common initial state;
(c) for each agent, obtaining an update that specifies, for one or more addresses of the state, a per-address numeric delta to be applied to the state, and obtaining a corresponding coefficient for the agent provided as a scalar, vector, or tensor, the coefficient being supplied by the agent or computed by a coordinator from agent-specific signals, and being replicated across one or more dimensions to align in shape with the agent's update when shapes differ;
(d) assembling a merged update by, for each address (a position in the state tensor):
(i) when exactly one agent provides a non-zero update at the address, setting the merged contribution equal to that update;
and (ii) when two or more agents provide non-zero updates at the address, combining the agents' per-address updates in accordance with their coefficients to produce a single per-address delta;
(e) when exactly one agent provides a non-zero update at the address, setting the merged contribution equal to that update;
and (i) when two or more agents provide non-zero updates at the address, combining the agents' per-address updates in accordance with their coefficients to produce a single per-address delta;
(f) updating the state by applying the merged update to produce an updated state for subsequent processing;
and (g) performing steps (c)-(e) at a coordination step of a processing algorithm that collects updates from the plurality of agents and combines them.

7 more elements

Dependencies are inferred from the claim text.

Persistent fixed-size memory for machine-learning using recurrent neural networks

Life of this patent

Persistent fixed-size memory for machine-learning using recurrent neural networks

Life of this patent

Abstract

Claims28