US 2026/0310378 A1 · Systems and Methods for Direct-From-Non-Volatile-Memory Neural Network Inference and Hidden-State Reuse

Abstract

Systems and methods for performing inference with a neural network model using model weights stored in non-volatile memory without loading the model weights into dynamic random-access memory (DRAM) prior to inference. A processing system reads model weights from non-volatile memory, streams the model weights to a compute engine, generates outputs by performing multiply-accumulate operations using the streamed model weights, and maintains intermediate states in on-chip volatile memory. In some embodiments, a recurrent computational model maintains a hidden state with fixed dimensionality independent of sequence length. In some embodiments, the processing system pre-processes context data to generate hidden states, stores the hidden states, and retrieves the hidden states to resume inference without reprocessing the context data. The hidden states may be stored as tuples comprising a key and a hidden state in a vector database.

Claims20

Dependent claims

A method performed by a processing system in a computing device, the processing system comprising a controller, a compute engine, an on-chip volatile memory, and a serial interface coupled to a non-volatile memory, the method comprising:

accessing, in the non-volatile memory, model weights of a neural network model in address ranges corresponding to respective layers of the neural network model;
streaming, by the controller through the serial interface and without loading first model weights into dynamic random-access memory (DRAM), first model weights for a first layer from a first address range of the non-volatile memory to the compute engine;
performing, by the compute engine, multiply-accumulate operations for the first layer using the first model weights;
streaming, by the controller through the serial interface and without loading second model weights into DRAM, second model weights for a second layer from a second address range of the non-volatile memory to the compute engine while the compute engine performs the multiply-accumulate operations for the first layer;
and
storing intermediate state data of the neural network model in the on-chip volatile memory during inference.

3 more elements

A non-transitory processor-readable medium having stored thereon data, configurations, or processor-readable instructions that, when used to control a state machine or when executed by a processing system in a computing device, cause the processing system to perform operations for direct-from-non-volatile-memory inference of a neural network model, the operations comprising:

accessing, in a non-volatile memory, model weights of the neural network model in address ranges corresponding to respective layers of the neural network model;
streaming, through a serial interface and without loading first model weights into DRAM, first model weights for a first layer from the non-volatile memory to a compute engine;
causing the compute engine to perform multiply-accumulate operations for the first layer using the first model weights;
streaming, through the serial interface and without loading second model weights into DRAM, second model weights for a second layer from the non-volatile memory to the compute engine while the multiply-accumulate operations for the first layer are being performed;
and
storing intermediate state data of the neural network model in on-chip volatile memory during inference.

3 more elements

Dependencies are inferred from the claim text.

Systems and Methods for Direct-From-Non-Volatile-Memory Neural Network Inference and Hidden-State Reuse

Life of this patent

Systems and Methods for Direct-From-Non-Volatile-Memory Neural Network Inference and Hidden-State Reuse

Life of this patent

Abstract

Claims20