Generating temporal sequences using diffusion transformer neural networks

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for generating an output temporal sequence of data elements conditioned on an input, the operations comprising:

• obtaining the input, wherein the input comprises a noise input comprising a plurality of latent representations for the output temporal sequence;
• updating each latent representation using a latent denoising neural network, the updating comprising:
• updating the latent representation at each of a plurality of iterations, comprising, at each iteration:
• processing an intermediate input for the iteration comprising the latent representation to generate a denoising output, the processing comprising:
• generating a sequence of tokens each representing a respective patch of the latent representation;
• generating an encoded representation of the sequence of tokens, the encoded representation comprising a sequence of input tokens;
• processing the encoded representation to generate a merged sequence of output tokens, wherein the merged sequence has a smaller number of output tokens than a number of input tokens in the sequence of input tokens;
• processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence that includes a respective updated output token for each of the output tokens in the merged sequence;
• processing the updated merged sequence to generate an expanded representation that includes a same number of tokens as the number of input tokens in the sequence of input tokens;
• and
• processing a decoder input derived from the expanded representation to generate the denoising output;
• updating the latent representation using the denoising output for the iteration;
• and
• generating the output temporal sequence of data elements by processing each of the updated latent representations using a decoder neural network.

A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating an output temporal sequence of data elements conditioned on an input, the operations comprising:

• obtaining the input, wherein the input comprises a noise input comprising a plurality of latent representations for the output temporal sequence;
• updating each latent representation using a latent denoising neural network, the updating comprising:
• updating the latent representation at each of a plurality of iterations, comprising, at each iteration:
• processing an intermediate input for the iteration comprising the latent representation to generate a denoising output, the processing comprising:
• generating a sequence of tokens each representing a respective patch of the latent representation;
• generating an encoded representation of the sequence of tokens, the encoded representation comprising a sequence of input tokens;
• processing the encoded representation to generate a merged sequence of output tokens, wherein the merged sequence has a smaller number of output tokens than a number of input tokens in the sequence of input tokens;
• processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence that includes a respective updated output token for each of the output tokens in the merged sequence;
• processing the updated merged sequence to generate an expanded representation that includes a same number of tokens as the number of input tokens in the sequence of input tokens;
• and
• processing a decoder input derived from the expanded representation to generate the denoising output;
• updating the latent representation using the denoising output for the iteration;
• and
• generating the output temporal sequence of data elements by processing each of the updated latent representations using a decoder neural network.