# US 2020/0394475 A1 · Method, System, and Computer Program Product for Generating New Items Compatible with Given Items

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394475 A1
- Document ID: US20200394475A1
- Status: Published application
- Kind: A1
- Application number: 16/439,861
- Filed: 2019-06-13
- Published: 2020-12-17
- Earliest claimed: 2019-06-13
- Assignees: Visa International Service Association
- Applicants: Visa International Service Association
- Inventors: Ablaikhan Akhazhanov; Maryam Moosaei; Hao Yang
- Web page: https://patentgrep.com/patent/US20200394475A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented method, a computing system, and a computer program product for generating new items compatible with given items may use data associated with a plurality of images and random noise data associated with a random noise image to train an adversarial network including a series of generator networks and a series of discriminator networks corresponding to the series of generator networks by modifying, using a loss function of the adversarial network that depends on a compatibility of the images, one or more parameters of the series of generator networks. The series of generator networks may generate a generated image associated with a generated item different than the given items.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining, with at least one processor, training data associated with a plurality of images and random noise data associated with a random noise image, wherein each image of the plurality of images is associated with a single item of a plurality of items;
training, with at least one processor, an adversarial network including a series of generator networks including a first generator network and at least one further generator network and a series of discriminator networks corresponding to the series of generator networks and including a first discriminator network and at least one further discriminator network by:
processing, with a first generator network of the series of generator networks, a representation of a set of images of the plurality of images and the random noise image, to generate generated image data associated with a generated image, wherein the generated image is associated with a generated item different than each item of the plurality of items;
processing, with a first discriminator network of the series of discriminator networks, the generated image data, to determine a prediction of whether the generated item is real;
processing, with each respective further generator network of the at least one further generator network, a further representation of the set of images of the plurality of images and the generated image data from a preceding generator network of the series of generator networks, to generate further generated image data associated with at least one further generated image, wherein the at least one further generated image is associated with the generated item different than each item of the plurality of items; and
processing, with each respective further discriminator network of the at least one further discriminator network, the further generated image data from a corresponding generator network of the series of generator networks, to determine at least one further prediction of whether the generated item is real; and
modifying, using a loss function of the adversarial network that depends on (i) a compatibility score including a prediction of a compatibility of the set of images and the generated image, (ii) at least one further compatibility score including a prediction of a compatibility of the set of images and the at least one further generated image, (iii) the prediction, and (iv) the at least one further prediction, one or more parameters of the series of generator networks.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein training the adversarial network includes:
processing, with the first generator network of the series of generator networks, (a) the representation of the set of images of the plurality of images and the random noise image and (b) a representation of an item category, to generate the generated image data associated with the generated image;
processing, with the first discriminator network of the series of discriminator networks, the generated image data and the representation of the item category, to determine the prediction, wherein the prediction includes predictions of (i) whether the generated item is real and (ii) whether the generated item is associated with the item category;
processing, with each respective further generator network of the at least one further generator network, (a) the at least one further representation of the set of images of the plurality of images and the generated image data from the preceding generator network of the series of generator networks and (b) the representation of the item category, to generate the further generated image data associated with the at least one further generated image; and
processing, with each respective further discriminator network of the at least one further discriminator network, the generated image data from the corresponding generator network of the series of generator networks and the representation of the item category, to determine the at least one further prediction, wherein the at least one further prediction includes predictions of (i) whether the generated item is real and (ii) whether the generated item is associated with the item category.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the set of images includes a subset of images of the plurality of images, and wherein the generated item is associated with a same item category as an item of the plurality of items not included in a set of items associated with the set of images.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein the set of images includes each image of the plurality of images, and wherein the generated item is associated with a different item category than each item of the plurality of items.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein training the adversarial network includes:
processing, with a first relation network of a series of relation networks corresponding to the series of generator networks and including the first relation network and at least one further relation network, the set of images of the plurality of images and the generated image data from the first generator network of the series of generator networks, to determine the compatibility score including the prediction of the compatibility of the set of images and the generated image;
processing, with each respective further relation network of the at least one further relation network, the set of images of the plurality of images and the further generated image data from a corresponding generator network of the series of generator networks, to determine the at least one further compatibility score including the prediction of the compatibility of the set of images and the at least one further generated image.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein training the adversarial network includes:
processing, with the first relation network of the series of relation networks, the set of images and the generated image, to determine a feature vector representation of the set of images and the generated image;
processing, with each respective further relation network of the at least one further relation network, the set of images and the at least one further generated image, to determine at least one further feature vector representation of the set of images and the at least one further generated image;
processing, with another relation network, the plurality of images, to determine another feature vector representation of the plurality of images;
determining a first divergence between the feature vector representation and the another feature vector representation;
determining at least one further divergence between each further feature vector representation of the at least one further feature vector representation and the another feature vector representation,
wherein the loss function of the adversarial network further depends on the divergence and the at least one further divergence.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein training the adversarial network includes:
processing, with an initial relation network, the set of images and the random noise image, to determine the representation of (a) the set of images of the plurality of images and (b) the random noise image as an initial feature vector representation of the set of images and the random noise image.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing, with at least one processor, the series of generator networks including the one or more parameters that have been modified based on the loss function of the adversarial network;
obtaining, with at least one processor, input data associated with a further plurality of images and further random noise data associated with a further random noise image; and
processing, with at least one processor using the series of generator networks, the input data and the further random noise data to generate output data.

### Claim 9

A computer-implemented method comprising:
obtaining, with at least one processor, input data associated with a plurality of images and random noise data associated with a random noise image, wherein each image of the plurality of images is associated with a single item of a plurality of items;
processing, with an initial relation network, a set of images of the plurality of images and the random noise image, to determine an initial feature vector representation of the set of images and the random noise image;
processing, with a first generator network of a series of generator networks including the first generator network and at least one further generator network, the initial feature vector representation, to generate image data associated with a generated image, wherein the generated image is associated with a generated item different than each item of the plurality of items;
processing, with each respective relation network of a series of relation networks corresponding to the series of generator networks, the set of images of the plurality of images and the generated image data from a corresponding generator network of the series of generator networks, to determine a further feature vector representation of the set of images and the generated image; and
processing, with each respective generator network of the at least one further generator network, the further feature vector representation from a preceding relation network of the series of relation networks, to generate further image data associated with at least one further generated image, wherein the at least one further generated image is associated with the generated item different than each item of the plurality of items.

### Claim 10 (depends on 9)

The computer implemented method of claim 9, further comprising:
processing, with the first generator network of the series of generator networks, the initial feature vector representation and a vector representation of an item category, to generate the image data associated with the generated image; and
processing, with each respective generator network of the at least one further generator network, the further feature vector representation from the preceding relation network of the series of relation networks and the representation of the item category, to generate the further image data associated with the at least one further generated image.

### Claim 11

A computing system comprising:
one or more processors programmed and/or configured to:
obtain training data associated with a plurality of images and random noise data associated with a random noise image, wherein each image of the plurality of images is associated with a single item of a plurality of items;
train an adversarial network including a series of generator networks including a first generator network and at least one further generator network and a series of discriminator networks corresponding to the series of generator networks and including a first discriminator network and at least one further discriminator network by:
process, with a first generator network of the series of generator networks, a representation of a set of images of the plurality of images and the random noise image, to generate generated image data associated with a generated image, wherein the generated image is associated with a generated item different than each item of the plurality of items;
process, with a first discriminator network of the series of discriminator networks, the generated image data, to determine a prediction of whether the generated item is real;
process, with each respective further generator network of the at least one further generator network, a further representation of the set of images of the plurality of images and the generated image data from a preceding generator network of the series of generator networks, to generate further generated image data associated with at least one further generated image, wherein the at least one further generated image is associated with the generated item different than each item of the plurality of items; and
process, with each respective further discriminator network of the at least one further discriminator network, the further generated image data from a corresponding generator network of the series of generator networks, to determine at least one further prediction of whether the generated item is real; and
modifying, using a loss function of the adversarial network that depends on (i) a compatibility score including a prediction of a compatibility of the set of images and the generated image, (ii) at least one further compatibility score including a prediction of a compatibility of the set of images and the at least one further generated image, (iii) the prediction, and (iv) the at least one further prediction, one or more parameters of the series of generator networks.

### Claim 12 (depends on 11)

The system of claim 11, wherein training the adversarial network includes:
processing, with the first generator network of the series of generator networks, (a) the representation of the set of images of the plurality of images and the random noise image and (b) a representation of an item category, to generate the generated image data associated with the generated image;
processing, with the first discriminator network of the series of discriminator networks, the generated image data and the representation of the item category, to determine the prediction, wherein the prediction includes predictions of (i) whether the generated item is real and (ii) whether the generated item is associated with the item category;
processing, with each respective further generator network of the at least one further generator network, (a) the at least one further representation of the set of images of the plurality of images and the generated image data from the preceding generator network of the series of generator networks and (b) the representation of the item category, to generate the further generated image data associated with the at least one further generated image; and
processing, with each respective further discriminator network of the at least one further discriminator network, the generated image data from the corresponding generator network of the series of generator networks and the representation of the item category, to determine the at least one further prediction, wherein the at least one further prediction includes predictions of (i) whether the generated item is real and (ii) whether the generated item is associated with the item category.

### Claim 13 (depends on 12)

The system of claim 12, wherein the set of images includes a subset of images of the plurality of images, and wherein the generated item is associated with a same item category as an item of the plurality of items not included in a set of items associated with the set of images.

### Claim 14 (depends on 12)

The system of claim 12, wherein the set of images includes each image of the plurality of images, and wherein the generated item is associated with a different item category than each item of the plurality of items.

### Claim 15 (depends on 11)

The system of claim 11, wherein training the adversarial network includes:
processing, with a first relation network of a series of relation networks corresponding to the series of generator networks and including the first relation network and at least one further relation network, the set of images of the plurality of images and the generated image data from the first generator network of the series of generator networks, to determine the compatibility score including the prediction of the compatibility of the set of images and the generated image;
processing, with each respective further relation network of the at least one further relation network, the set of images of the plurality of images and the further generated image data from a corresponding generator network of the series of generator networks, to determine the at least one further compatibility score including the prediction of the compatibility of the set of images and the at least one further generated image.

### Claim 16 (depends on 11)

The system of claim 11, wherein training the adversarial network includes:
processing, with the first relation network of the series of relation networks, the set of images and the generated image, to determine a feature vector representation of the set of images and the generated image;
processing, with each respective further relation network of the at least one further relation network, the set of images and the at least one further generated image, to determine at least one further feature vector representation of the set of images and the at least one further generated image;
processing, with another relation network, the plurality of images, to determine another feature vector representation of the plurality of images;
determining a first divergence between the feature vector representation and the another feature vector representation;
determining at least one further divergence between each further feature vector representation of the at least one further feature vector representation and the another feature vector representation,
wherein the loss function of the adversarial network further depends on the divergence and the at least one further divergence.

### Claim 17 (depends on 11)

The system of claim 11, wherein training the adversarial network includes:
processing, with an initial relation network, the set of images and the random noise image, to determine the representation of (a) the set of images of the plurality of images and (b) the random noise image as an initial feature vector representation of the set of images and the random noise image.

### Claim 18 (depends on 11)

The system of claim 11, wherein the one or more processors are further programmed and/or configured to:
provide the series of generator networks including the one or more parameters that have been modified based on the loss function of the adversarial network;
obtain input data associated with a further plurality of images and further random noise data associated with a further random noise image; and
process, using the series of generator networks, the input data and the further random noise data to generate output data.

### Claim 19

A computing system comprising:
one or more processors programmed and/or configured to:
obtain input data associated with a plurality of images and random noise data associated with a random noise image, wherein each image of the plurality of images is associated with a single item of a plurality of items;
process, with an initial relation network, a set of images of the plurality of images and the random noise image, to determine an initial feature vector representation of the set of images and the random noise image;
process, with a first generator network of a series of generator networks including the first generator network and at least one further generator network, the initial feature vector representation, to generate image data associated with a generated image, wherein the generated image is associated with a generated item different than each item of the plurality of items;
process, with each respective relation network of a series of relation networks corresponding to the series of generator networks, the set of images of the plurality of images and the generated image data from a corresponding generator network of the series of generator networks, to determine a further feature vector representation of the set of images and the generated image; and
process, with each respective generator network of the at least one further generator network, the further feature vector representation from a preceding relation network of the series of relation networks, to generate further image data associated with at least one further generated image, wherein the at least one further generated image is associated with the generated item different than each item of the plurality of items.

### Claim 20 (depends on 19)

The computing system of claim 19, wherein the one or more processors are further programmed and/or configured to:
process, with the first generator network of the series of generator networks, the initial feature vector representation and a vector representation of an item category, to generate the image data associated with the generated image; and
process, with each respective generator network of the at least one further generator network, the further feature vector representation from the preceding relation network of the series of relation networks and the representation of the item category, to generate the further image data associated with the at least one further generated image.
