US 2025/0053786 A1 · GENERATING DISCRETE DATA USING DIFFUSION NEURAL NETWORKS

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a network output of high dimensional data comprising one or more output tokens. In one aspect, a system comprises a neural network system configured to initialize an analog bit representation of the network output comprising a set of continuous numeric values for each of the output tokens. The neural network system generates an updated analog bit representation that comprises a set of updated continuous numeric values. At each of a plurality of update iterations, the neural network system processes a diffusion input comprising the analog bit representation using a diffusion machine learning model to update the analog bit representation.

Claims20

Dependent claims

A method performed by one or more data processing apparatus for generating a network output comprising one or more output tokens, the method comprising:

initializing an analog bit representation of the network output that comprises a respective set of continuous numeric values for each of the output tokens;
generating an updated analog bit representation that comprises a respective set of updated continuous numeric values for each of the output tokens, the generating comprising, at each of a plurality of update iterations:
processing a diffusion input comprising the analog bit representation using a diffusion machine learning model to update the analog bit representation;
and
for each output token:
generating a binary representation of the output token by generating, from each updated continuous value in the set of respective continuous values for the output token, a corresponding binary value;
and
generating the output token by decoding the binary representation of the output token.

5 more elements

A system comprising:

one or more computers;
and
one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
initializing an analog bit representation of the network output that comprises a respective set of continuous numeric values for each of the output tokens;
generating an updated analog bit representation that comprises a respective set of updated continuous numeric values for each of the output tokens, the generating comprising, at each of a plurality of update iterations:
processing a diffusion input comprising the analog bit representation using a diffusion machine learning model to update the analog bit representation;
and
for each output token:
generating a binary representation of the output token by generating, from each updated continuous value in the set of respective continuous values for the output token, a corresponding binary value;
and
generating the output token by decoding the binary representation of the output token.

8 more elements

One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

initializing an analog bit representation of the network output that comprises a respective set of continuous numeric values for each of the output tokens;
generating an updated analog bit representation that comprises a respective set of updated continuous numeric values for each of the output tokens, the generating comprising, at each of a plurality of update iterations:
processing a diffusion input comprising the analog bit representation using a diffusion machine learning model to update the analog bit representation;
and
for each output token:
generating a binary representation of the output token by generating, from each updated continuous value in the set of respective continuous values for the output token, a corresponding binary value;
and
generating the output token by decoding the binary representation of the output token.

5 more elements

Dependencies are inferred from the claim text.

Generating discrete data using diffusion neural networks

Life of this patent

Generating discrete data using diffusion neural networks

Life of this patent

Abstract

Claims20