Data processing method and apparatus, electronic device and storage medium

A data processing apparatus, comprising:

• an inference unit, configured to perform an inference operation on data to be processed through a neural network,
• wherein the inference operation comprises:
• using the neural network to perform M rounds of inference successively on the data to be processed, wherein the data to be processed comprises a plurality of batches of input data, each batch of the input data comprises N1 input data elements, and an entire inference process is completed on only one batch of the input data in each round of the inference, and M and N1 are positive integers greater than 1;
• the M rounds of inference comprise an i-th round of inference and an (i+1)-th round of inference executed successively in order, i is a positive integer and less than M,
• wherein an inference process of the i-th round of inference comprises:
• at a first stage of the inference process of the i-th round of inference, performing inference on N2 input data elements,
• at a second stage of the inference process of the i-th round of inference, performing inference on N1 input data elements to complete the entire inference process on the N1 input data elements belonging to a same batch of the input data;
• wherein the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of the input data are a subset of the N2 input data elements;
• an inference process of the (i+1)-th round of inference comprises:
• for N2-N1 input data elements for which the inference is incomplete in the i-th round of inference but for which output tokens have been acquired through the inference, performing inference on the N2-N1 input data elements while taking into account the output tokens.