# US 12,001,520 B2 · Generating simulated images that enhance socio-demographic diversity > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,520 B2 - Document ID: US12001520B2 - Status: Granted - Kind: B2 - Application number: 17/485,780 - Filed: 2021-09-27 - Published: 2023-03-30 - Granted: 2024-06-04 - Earliest claimed: 2021-09-27 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Ritwik Sinha; Sridhar Mahadevan; Moumita Sinha; Md Mehrab Tanjim; Krishna Kumar Singh; David Arbour - Web page: https://patentgrep.com/patent/US12001520B2 ## CPC - G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries - G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/094: Adversarial learning - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 40/172: {Classification, e.g. identification} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods and systems disclosed herein relate generally to systems and methods for generating simulated images for enhancing socio-demographic diversity. An image-generating application receives a request that includes a set of target socio-demographic attributes. The set of target socio-demographic attributes can define a gender, age, and/or race of a subject that are non-stereotypical for a particular occupation. The image-generating application applies the a machine-learning model to the set of target socio-demographic attributes. The machine-learning model generates a simulated image depicts a subject having visual characteristics that are defined by the set of target socio-demographic attributes. ## Claims ### Claim 1 A computer-implemented method of generating simulated images that enhance socio-demographic diversity, the method comprising: receiving, by an image-generating application, a request to generate a simulated image of a subject, wherein the request includes a set of target socio-demographic attributes that define one or more visual characteristics of the subject; applying, by the image-generating application, a machine-learning model to the set of target socio-demographic attributes to generate the simulated image, wherein applying the machine-learning model includes: processing, by a generator component of the machine-learning model, an embedding representing the set of target socio-demographic attributes to generate a candidate simulated image depicting a candidate subject represented by one or more candidate visual characteristics; determining, by a discriminator component of the machine-learning model, that the one or more candidate visual characteristics of the candidate simulated image depict one or more anatomical features; generating, by the discriminator component, a set of predicted sociodemographic attributes that define the one or more candidate visual characteristics; determining, by the discriminator component, whether the set of predicted socio-demographic attributes substantially match the set of target socio-demographic attributes; and in response to determining that the set of predicted socio-demographic attributes substantially match the set of target socio-demographic attributes, classifying, by the discriminator component, the candidate simulated image as the simulated image; and outputting, by the image-generating application, the simulated image. ### Claim 2 (depends on 1) The method of claim 1, wherein applying the machine-learning model to the set of target socio-demographic attributes includes: generating, by the image-generating application, a feature vector that represents the set of target socio-demographic attributes; accessing, by the image-generating application, a latent code; applying, by the image-generating application, one or more neural networks to the feature vector and the latent code to generate the embedding, wherein the machine-learning model is applied to the embedding to generate the simulated image. ### Claim 3 (depends on 1) The method of claim 1, wherein determining whether the set of predicted socio-demographic attributes substantially match the set of target socio-demographic attributes includes: determining a loss between the set of predicted socio-demographic attributes and the set of target socio-demographic attributes; and modifying one or more parameters of the discriminator component based on the determined loss. ### Claim 4 (depends on 3) The method of claim 3, wherein determining the loss includes using a modified loss function of the discriminator component, wherein the modified loss function is configured to increase a probability of the candidate simulated image to depict the candidate subject having the one or more visual characteristics. ### Claim 5 (depends on 1) The method of claim 1, wherein the machine-learning model was trained using a training dataset comprising a plurality of training images, wherein the plurality of training images include one or more duplicate copies of a training image being associated with the set of target socio-demographic attributes. ### Claim 6 (depends on 1) The method of claim 1, wherein the machine-learning model was trained using a training dataset comprising a plurality of training images, wherein the plurality of training images include one or more augmented copies of a training image being associated with the set of target socio-demographic attributes. ### Claim 7 (depends on 6) The method of claim 6, wherein an augmented copy of the one or more augmented copies is generated by modifying a color of the training image being associated with the set of target socio-demographic attributes. ### Claim 8 (depends on 6) The method of claim 6, wherein an augmented copy of the one or more augmented copies is generated by modifying an orientation of the training image being associated with the set of target socio-demographic attributes. ### Claim 9 (depends on 1) The method of claim 1, wherein the machine-learning model is a generative adversarial network. ### Claim 10 (depends on 1) The method of claim 1, wherein the set of target socio-demographic attributes include age, gender, race, and/or an occupation of the subject. ### Claim 11 A system comprising: a data-sampling engine configured to: receive a plurality of images, wherein each image of the plurality of images depict a respective subject; and generate a set of training images from the plurality of images, wherein the set of training images include one or more duplicate copies of a training image being associated with a set of target socio-demographic attributes; a data-augmentation engine configured to: receive the set of training images from the data-sampling engine; and generate, based on the set of training images, a set of augmented training images, wherein the set of augmented training images include one or more augmented copies of each training image of the set of training images; and a training engine configured to: receive the set of augmented training images from the data-augmentation engine; process each augmented training image of the set of augmented training images to train a machine-learning model to generate a simulated image, wherein the simulated image depicts a subject that has one or more visual characteristics defined by the set of target socio-demographic attributes, wherein training the machine-learning model comprises: generating an embedding of each augmented training image; and using the embedding to train a generator component of the machine-learning model to generate a candidate simulated image depicting a candidate subject represented by one or more candidate visual characteristics; and output the trained machine-learning model. ### Claim 12 (depends on 11) The system of claim 11, wherein the training engine is further configured to train the machine-learning model by: training a discriminator component of the machine-learning model to: receive, from the generator component, the candidate simulated image; determine that the one or more candidate visual characteristics depict one or more anatomical features; generate a set of predicted socio-demographic attributes that define the one or more candidate visual characteristics; determine whether the set of predicted socio-demographic attributes substantially match the set of target socio-demographic attributes; and in response to determining that the set of predicted socio-demographic attributes substantially match the set of target socio-demographic attributes, classify the candidate simulated image as the simulated image. ### Claim 13 (depends on 11) The system of claim 11, wherein the embedding is generated by: generating a feature vector that represents the augmented training image; accessing a latent code; and applying one or more neural networks to the feature vector and the latent code to generate the embedding. ### Claim 14 (depends on 11) The system of claim 11, wherein the machine-learning model includes a modified loss function configured to increase a probability of the simulated image depicting the subject having the one or more visual characteristics defined by the set of target sociodemographic attributes. ### Claim 15 (depends on 11) The system of claim 11, wherein the machine-learning model is a generative adversarial network. ### Claim 16 (depends on 11) The system of claim 11, wherein an augmented training image of the set of augmented training images is generated by modifying a color of a corresponding training image of the set of training images. ### Claim 17 (depends on 11) The system of claim 11, wherein an augmented training image of the set of augmented training images is generated by modifying an orientation of a corresponding training image of the set of training images. ### Claim 18 A computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform actions including: receiving a request to generate a simulated image of a subject, wherein the request includes a set of target socio-demographic attributes that define one or more visual characteristics of the subject; a step for generating the simulated image of the subject, wherein the simulated image is generated by: (i) applying a generator component of a machine-learning model to the set of target socio-demographic attributes to process an embedding representing the set of target socio-demographic attributes to generate a candidate simulated image depicting a candidate subject represented by one or more candidate visual characteristics; and (ii) applying a discriminator component of the machine-learning model to the candidate simulated image to generate a set of predicted socio-demographic attributes of the candidate simulated image, determine that the set of predicted socio-demographic attributes substantially match the set of target socio-demographic attributes, and classify the candidate simulated image as the simulated image; and outputting the simulated image. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein the set of target socio-demographic attributes include age, gender, race, and/or an occupation of the subject.