Graphic rendering using diffusion models
AssigneeDream3D, Inc.
InventorTony Francis
Life of this patent
Estimating the term…
Abstract
A system or method for generating computer graphics. One or more three-dimensional (3D) scenes are obtained and rasterized into a first set of two-dimensional (2D) images having a first resolution. Features are extracted from the first set of 2D images, and text prompts are generated based on the features. A diffusion model is applied to the features, and the text prompts to generate a second set of 2D images having a second resolution greater than the first resolution. The diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt. The second set of 2D images having the second resolution are caused to be rendered at a client device.
Claims20
A method comprising:
- generating one or more embeddings and one or more sets of latent variables based on features extracted from a rasterized sequence of one or more three-dimensional scenes;
- sampling a first set of one or more two-dimensional images of a first resolution at different times;
- applying a diffusion model to 1) features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more embeddings to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding embeddings to generate an image consistent with an embedding, wherein the diffusion model includes a visual transformer model configured to receive the features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the one or more embeddings, and the one or more sets of latent variables to generate the second set of one or more images;
- and
- causing the second set of one or more images to be rendered at a client device.
A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
- generating one or more embeddings and one or more sets of latent variables based on features extracted from a rasterized sequence of one or more three-dimensional scenes;
- sampling a first set of one or more two-dimensional images of a first resolution at different times;
- applying a diffusion model to 1) features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more embeddings to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding embeddings to generate an image consistent with an embedding, wherein the diffusion model includes a visual transformer model configured to receive the features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the one or more embeddings, and the one or more sets of latent variables to generate the second set of one or more images;
- and
- causing the second set of one or more images to be rendered at a client device.
A computer system comprising:
- one or more processors;
- and
- a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:
- generating one or more embeddings and one or more sets of latent variables based on features extracted from a rasterized sequence of one or more three-dimensional scenes;
- sampling a first set of one or more two-dimensional images of a first resolution at different times;
- applying a diffusion model to 1) features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more embeddings to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding embeddings to generate an image consistent with an embedding, wherein the diffusion model includes a visual transformer model configured to receive the features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the one or more embeddings, and the one or more sets of latent variables to generate the second set of one or more images;
- and
- causing the second set of one or more images to be rendered at a client device.
Dependencies are inferred from the claim text.