Generating temporal sequences using diffusion transformer neural networks

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for generating an output temporal sequence of data elements conditioned on an input, the operations comprising:

• obtaining the input, wherein the input comprises a noise input comprising a respective latent representation for each of a plurality of segments of the temporal sequence, each segment comprising one or more data elements of the output temporal sequence of data elements;
• updating, for each segment, the latent representation for the segment using a latent denoising neural network, wherein the latent denoising neural network is configured to autoregressively update the latent representations for each segment, the updating comprising, for each segment other than the first segment:
• obtaining a memory vector representing one or more hidden states generated by the latent denoising neural network when updating the latent representations for one or more preceding segments;
• updating the latent representation for the segment at each of a plurality of iterations, wherein the updating comprises, at each iteration:
• processing an intermediate input for the iteration comprising the latent representation and the memory vector to generate a denoising output;
• and
• updating the latent representation using the denoising output for the iteration;
• and
• generating the output temporal sequence of data elements by processing the latent representations for the plurality of segments using a decoder neural network.

A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating an output temporal sequence of data elements conditioned on an input, the operations comprising:

• obtaining the input, wherein the input comprises a noise input comprising a respective latent representation for each of a plurality of segments of the temporal sequence, each segment comprising one or more data elements of the output temporal sequence of data elements;
• updating, for each segment, the latent representation for the segment using a latent denoising neural network, wherein the latent denoising neural network is configured to autoregressively update the latent representations for each segment, the updating comprising, for each segment other than the first segment:
• obtaining a memory vector representing one or more hidden states generated by the latent denoising neural network when updating the latent representations for one or more preceding segments;
• updating the latent representation for the segment at each of a plurality of iterations, wherein the updating comprises, at each iteration:
• processing an intermediate input for the iteration comprising the latent representation and the memory vector to generate a denoising output;
• and
• updating the latent representation using the denoising output for the iteration;
• and
• generating the output temporal sequence of data elements by processing the latent representations for the plurality of segments using a decoder neural network.