# US 12,602,743 B1 · Image synthesis using one or more neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,743 B1 - Document ID: US12602743B1 - Status: Granted - Kind: B1 - Application number: 16/443,664 - Filed: 2019-06-17 - Granted: 2026-04-14 - Earliest claimed: 2019-06-17 - Assignees: NVIDIA Corporation - Applicants: Nvidia Corporation - Inventors: Ziyue Xu; Xiaosong Wang; Ling Zhang; Holger Roth; Daguang Xu; Fausto Milletari; Hoo Chang Shin; Dong Yang - Web page: https://patentgrep.com/patent/US12602743B1 ## CPC - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06F 7/57: Arithmetic logic units [ALU], i.e. arrangements or devices for performing two or more of the operations covered by groups G06F7/483 – G06F7/556 or for performing logical operations {(G06F7/49, G06F7/491 take precedence)} - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/048: Activation functions - G06N 3/063: using electronic means - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 5/04: Inference or reasoning models - G06T 7/11: Region-based segmentation - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Apparatuses, systems, and techniques to infer synthesized images including digital representations of objects blended realistically with appropriate background images. In at least one embodiment, background image data and a description of semantic features for a type of object are fused together to infer such a synthesized image using one or more neural networks. ## Claims ### Claim 1 One or more processors, comprising: circuitry to: receive, as input to one or more neural networks, one or more user-generated descriptions of semantic features associated with one or more objects; and cause the one or more neural networks to generate one or more images depicting the one or more objects, wherein one or more pixels of the one or more images depicting at least a portion of the one or more objects are blended with pixels of one or more background images based, at least in part, on the one or more user-generated descriptions of the semantic features associated with the one or more objects. ### Claim 2 (depends on 1) The one or more processors of claim 1, wherein the circuitry is further to be configured to: blend a portion of the one or more objects with the pixels of the one or more background images at a plurality of resolution levels using two or more layers of the one or more neural networks. ### Claim 3 (depends on 1) The one or more processors of claim 1, wherein the one or more user-generated descriptions of the semantic features includes a set of semantic feature values, a semantic feature vector, a semantic feature matrix, a list of semantic feature attributes, or a textual description of the semantic features. ### Claim 4 (depends on 1) The one or more processors of claim 1, wherein the circuitry is further to be configured to: accept, as input, background image data and the one or more user-generated descriptions of the semantic features, the one or more user-generated descriptions of the semantic features associated with visual attributes of the one or more objects; and infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using image data and semantic feature description data. ### Claim 5 (depends on 4) The one or more processors of claim 4, wherein the GAN was trained in part by transforming the one or more user-generated descriptions of the semantic features to an image feature vector and fusing the image feature vector with an encoding of the image data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of an object blended with a background portion of the image data. ### Claim 6 (depends on 5) The one or more processors of claim 5, wherein the GAN was further trained by passing the synthetic image, the segmentation mask, and the image feature vector to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values. ### Claim 7 A system comprising: one or more processors to: receive, as input to one or more neural networks, one or more user-generated descriptions of semantic features associated with one or more objects; store the one or more user-generated descriptions of the semantic features associated with the one or more objects; and cause the one or more neural networks to generate one or more images depicting the one or more objects, wherein one or more pixels of the one or more images depicting at least a portion of the one or more objects are blended with pixels of one or more background images based, at least in part, on the one or more user-generated descriptions of the semantic features. ### Claim 8 (depends on 7) The system of claim 7, wherein the one or more processors are further to be configured to: blend a portion of the one or more objects with the pixels of the one or more background images at a plurality of resolution levels using two or more layers of the one or more neural networks. ### Claim 9 (depends on 7) The system of claim 7, wherein the one or more user-generated descriptions of the semantic features includes a set of semantic feature values, a semantic feature vector, a semantic feature matrix, a list of semantic feature attributes, or a textual description of the semantic features. ### Claim 10 (depends on 7) The system of claim 7, wherein the one or more processors are further to be configured to: accept, as input, background image data and the one or more user-generated descriptions of the semantic features, the one or more user-generated descriptions of the semantic features associated with visual attributes of the one or more objects; and infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using image data and semantic feature description data. ### Claim 11 (depends on 10) The system of claim 10, wherein the GAN was trained in part by transforming the one or more user-generated descriptions of the semantic features to an image feature vector and fusing the image feature vector with an encoding of the image data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of an object blended with a background portion of the image data. ### Claim 12 (depends on 11) The system of claim 11, wherein the GAN was further trained by passing the synthetic image, the segmentation mask, and the image feature vector to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values. ### Claim 13 A method comprising: receiving, as input to one or more neural networks, one or more user-generated descriptions of semantic features associated with one or more objects; causing the one or more neural networks to generate one or more images depicting the one or more objects, wherein one or more pixels of the one or more images depicting at least a portion of the one or more objects are blended with pixels of one or more background images based, at least in part, on the one or more user-generated descriptions of the semantic features associated with the one or more objects; and storing the one or more images. ### Claim 14 (depends on 13) The method of claim 13, further comprising: blending a portion of the one or more objects with the pixels of the one or more background images at a plurality of resolution levels using two or more layers of the one or more neural networks. ### Claim 15 (depends on 13) The method of claim 13, wherein the one or more user-generated descriptions of the semantic features includes a set of semantic feature values, a semantic feature vector, a semantic feature matrix, a list of semantic feature attributes, or a textual description of the semantic features. ### Claim 16 (depends on 13) The method of claim 13, further comprising: accepting, as input, background image data and the one or more user-generated descriptions of the semantic features, the one or more user-generated descriptions of the semantic features associated with visual attributes of the one or more objects; and inferring the one or more images using a multi-conditional generative adversarial network (GAN) trained using image data and semantic feature description data. ### Claim 17 (depends on 16) The method of claim 16, wherein the GAN was trained in part by transforming the one or more user-generated descriptions of the semantic features to an image feature vector and fusing the image feature vector with an encoding of the image data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of an object blended with a background portion of the image data. ### Claim 18 (depends on 17) The method of claim 17, wherein the GAN was further trained by passing the synthetic image, the segmentation mask, and the image feature vector to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values. ### Claim 19 One or more processors, comprising: circuitry to: receive, as input to one or more neural networks, one or more user-generated descriptions of semantic features associated with one or more objects; and train the one or more neural networks to generate one or more images depicting the one or more objects, wherein one or more pixels of the one or more images depicting at least a portion of the one or more objects are blended with pixels of one or more background images based, at least in part, on a the one or more user-generated descriptions of the semantic features associated with the one or more objects. ### Claim 20 (depends on 19) The one or more processors of claim 19, wherein the circuitry is further to be configured to train the one or more neural networks to blend a portion of the one or more objects with the pixels of the one or more background images at a plurality of resolution levels using two or more layers of the one or more neural networks. ### Claim 21 (depends on 19) The one or more processors of claim 19, wherein the one or more user-generated descriptions of the semantic features includes a set of semantic feature values, a semantic feature vector, a semantic feature matrix, a list of semantic feature attributes, or a textual description of the semantic features. ### Claim 22 (depends on 19) The one or more processors of claim 19, wherein the circuitry further to be configured to train the one or more neural networks to: accept, as input, background image data and the one or more user-generated descriptions of the semantic features, the one or more user-generated descriptions of the semantic features associated with visual attributes of the one or more objects; and infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using image data and semantic feature description data. ### Claim 23 (depends on 22) The one or more processors of claim 22, wherein the GAN is to be trained in part by transforming the semantic features description data to an image feature vector and fusing the image feature vector with an encoding of the image data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of an object blended with a background portion of the image data. ### Claim 24 (depends on 23) The one or more processors of claim 23, wherein the GAN is further to be trained by passing the synthetic image, the segmentation mask, and the image feature vector to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values. ### Claim 25 A system, comprising: one or more memories to store one or more user-generated descriptions of semantic features associated with one or more objects; and one or more processors to train one or more neural networks to generate one or more images depicting the one or more objects, wherein one or more pixels of the one or more images depicting at least a portion of the one or more objects are blended with pixels of one or more background images based, at least in part, on the one or more user-generated descriptions of the semantic features associated with the one or more objects. ### Claim 26 (depends on 25) The system of claim 25, wherein the one or more processors are further to be configured to train the one or more neural networks to blend a portion of the one or more objects with the pixels of the one or more background images at a plurality of resolution levels using two or more layers of the one or more neural networks. ### Claim 27 (depends on 25) The system of claim 25, wherein the one or more user-generated descriptions of the semantic features includes a set of semantic feature values, a semantic feature vector, a semantic feature matrix, a list of semantic feature attributes, or a textual description of the semantic features. ### Claim 28 (depends on 25) The system of claim 25, wherein the one or more processors are further to be configured to train the one or more neural networks to: accept, as input, background image data and the one or more user-generated descriptions of the semantic features, the one or more user-generated descriptions of the semantic features associated with visual attributes of the one or more objects; and infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using image data and semantic feature description data. ### Claim 29 (depends on 28) The system of claim 28, wherein the GAN is to be trained in part by transforming the semantic features description data to an image feature vector and fusing the image feature vector with an encoding of the image data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of an object blended with a background portion of the image data. ### Claim 30 (depends on 29) The system of claim 29, wherein the GAN is further to be trained by passing the synthetic image, the segmentation mask, and the image feature vector to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values. ### Claim 31 A method comprising: receiving, as input to one or more neural networks, one or more user-generated descriptions of semantic features associated with one or more objects; training the one or more neural networks to generate one or more images depicting the one or more objects, wherein one or more pixels of the one or more images depicting at least a portion of the one or more objects are blended with pixels of one or more background images based, at least in part, on the one or more user-generated descriptions of the semantic features associated with the one or more objects; and storing the one or more neural networks. ### Claim 32 (depends on 31) The method of claim 31, further comprising: blending a portion of the one or more objects with the pixels of the one or more background images at a plurality of resolution levels using two or more layers of the one or more neural networks. ### Claim 33 (depends on 31) The method of claim 31, wherein the one or more user-generated descriptions of the semantic features includes a set of semantic feature values, a semantic feature vector, a semantic feature matrix, a list of semantic feature attributes, or a textual description of the semantic features. ### Claim 34 (depends on 31) The method of claim 31, further comprising training the one or more neural networks to: accept, as input, background image data and the one or more user-generated descriptions of the semantic features, the one or more user-generated descriptions of the semantic features associated with visual attributes of the one or more objects; and infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using image data and semantic feature description data. ### Claim 35 (depends on 34) The method of claim 34, wherein the GAN is to be trained in part by transforming the semantic features description data to an image feature vector and fusing the image feature vector with an encoding of the image data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of an object blended with a background portion of the image data. ### Claim 36 (depends on 35) The method of claim 35, wherein the GAN is further to be trained by passing the synthetic image, the segmentation mask, and the image feature vector to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values.