# US 2021/0342701 A1 · Deep learning based visual compatibility prediction for bundle recommendations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342701 A1
- Document ID: US20210342701A1
- Status: Published application
- Kind: A1
- Application number: 16/865,572
- Filed: 2020-05-04
- Published: 2021-11-04
- Earliest claimed: 2020-05-04
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Kumar AYUSH; Ayush CHOPRA; Patel Utkarsh GOVIND; Balaji KRISHNAMURTHY; Anirudh SINGHAL
- Web page: https://patentgrep.com/patent/US20210342701A1

## CPC

- G06Q 30/0631: {Recommending goods or services}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2193: {based on specific statistical tests}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of the present invention provide systems, methods, and computer storage media for predicting visual compatibility between a bundle of catalog items (e.g., a partial outfit) and a candidate catalog item to add to the bundle. Visual compatibility prediction may be jointly conditioned on item type, context, and style by determining a first compatibility score jointly conditioned on type (e.g., category) and context, determining a second compatibility score conditioned on outfit style, and combining the first and second compatibility scores into a unified visual compatibility score. A unified visual compatibility score may be determined for each of a plurality of candidate items, and the candidate item with the highest unified visual compatibility score may be selected to add to the bundle (e.g., fill the in blank for the partial outfit).

## Claims

### Claim 1

One or more computer storage media storing computer-usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
generating, by a first neural network comprising an autoencoder with a graph neural network, a first compatibility score that quantifies compatibility between a partial outfit of items and a candidate item to add to the partial outfit, and is jointly conditioned on item category and item context;
generating, by a second neural network comprising a style autoencoder, a second compatibility score that quantifies visual compatibility between style of the partial outfit and style of the candidate item;
using a learned transformation function to combine the first compatibility score and the second compatibility score into a unified visual compatibility score; and
performing one or more pre-determined actions based on the unified visual compatibility score.

### Claim 2 (depends on 1)

The one or more computer storage media of claim 1, wherein generating the first compatibility score further comprises:
representing catalog items, including the items in the partial outfit and the candidate item, as nodes of an incomplete graph with edges connecting nodes corresponding to the items in the partial outfit; and
using the autoencoder with the graph neural network to predict probabilities of missing edges in the incomplete graph.

### Claim 3 (depends on 1)

The one or more computer storage media of claim 1, wherein generating the first compatibility score further comprises:
using the autoencoder with the graph neural network to predict pairwise similarities quantifying similarity between pairs of catalog items, the catalog items including the items in the partial outfit and the candidate item; and
generating the first compatibility score by averaging a set of the pairwise similarities between the candidate item and each of the items from the partial outfit.

### Claim 4 (depends on 1)

The one or more computer storage media of claim 1, wherein generating the first compatibility score further comprises:
representing items of a catalog, including the items in the partial outfit and the candidate item, as an incomplete graph; and
using the autoencoder with the graph neural network to predict missing edges in the incomplete graph based on a corresponding adjacency matrix that is weighted by co-occurrence, within the catalog, of categories of items corresponding to pairs of connected nodes.

### Claim 5 (depends on 1)

The one or more computer storage media of claim 1, wherein generating the second compatibility score is further based at least in part on a style embedding for the partial outfit generated by attending over style embeddings for each of the items in the partial outfit.

### Claim 6 (depends on 1)

The one or more computer storage media of claim 1, wherein generating the second compatibility score further comprises:
generating, for each of the items in the partial outfit, an outfit style attention based on pairwise similarities predicted by the first neural network; and
generating, using the second neural network, a style embedding for the partial outfit based on the outfit style attention for each of the items in the partial outfit.

### Claim 7 (depends on 1)

The one or more computer storage media of claim 1, wherein generating the second compatibility score further comprises:
using the style autoencoder to predict a first outfit mixture ratio in a style basis corresponding to the partial outfit;
using the style autoencoder to predict a second outfit mixture ratio in the style basis corresponding to the candidate item being added to the partial outfit; and
generating the second compatibility score based on a decrease in uncertainty from the first outfit mixture ratio to the second outfit mixture ratio.

### Claim 8 (depends on 1)

The one or more computer storage media of claim 1, further comprising learning the transformation function using a recurrent neural network controller configured to repetitively predict components of a core unit to form a composite function that combines and weights the first compatibility score and the second compatibility score.

### Claim 9 (depends on 1)

The one or more computer storage media of claim 1, wherein the one or more pre-determined actions comprise causing a presentation of the unified visual compatibility score or adding the candidate item to the partial outfit.

### Claim 10

A computerized method for visual compatibility prediction, the method comprising:
for each of a plurality of candidate items, of a catalog, to fill in a blank in a bundle of items of the catalog, generating a compatibility score quantifying compatibility between the candidate item and the bundle, at least in part by:
representing the items in the bundle as connected nodes in an incomplete graph; and
using an autoencoder with a graph neural network to predict missing edges in the incomplete graph based on a corresponding adjacency matrix that is weighted by co-occurrence, within the catalog, of categories of items corresponding to pairs of the connected nodes; and
performing at least one of causing a presentation of a first candidate item having the highest compatibility score or adding the first candidate item to the bundle.

### Claim 11 (depends on 10)

The method of claim 10,
wherein the autoencoder is configured to predict pairwise similarities quantifying similarity between pairs of items in the catalog, and
wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle further comprises generating a first compatibility score for the candidate item by averaging a set of the pairwise similarities between the candidate item and each of the items in the bundle.

### Claim 12 (depends on 10)

The method of claim 10, wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle is further based at least in part on a style embedding for the bundle generated by attending over style embeddings for each of the items in bundle.

### Claim 13 (depends on 10)

The method of claim 10, wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle further comprises:
generating, for each of the items in the bundle, a bundle style attention based on pairwise similarities predicted by a first neural network comprising the autoencoder with the graph neural network; and
generating, using a second neural network comprising a style autoencoder, a style embedding for the bundle based on the outfit style attention for each of the items in the bundle.

### Claim 14 (depends on 10)

The method of claim 10, wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle further comprises:
using a style autoencoder, distinct from the autoencoder with the graph neural network, to predict a first outfit mixture ratio in a style basis corresponding to the bundle;
using the style autoencoder to predict a second outfit mixture ratio in the style basis corresponding to the candidate item being added to the bundle; and
generating a second compatibility score based on a decrease in uncertainty from the first outfit mixture ratio to the second outfit mixture ratio.

### Claim 15 (depends on 10)

The method of claim 10, wherein the compatibility score quantifying compatibility between the candidate item and the bundle is a unified visual compatibility score generated by applying a learned transformation function to combine and weight a first compatibility score and a second compatibility score.

### Claim 16 (depends on 10)

The method of claim 10, the method further comprising:
learning a transformation function using a recurrent neural network controller configured to repetitively predict components of a core unit to form a composite function that combines and weights a first compatibility score and a second compatibility score; and
applying the transformation function to generate the compatibility score for each of the candidate items.

### Claim 17

A computer system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors; and
a means for generating a recommendation to add one of a plurality of candidate items from a catalog to an outfit of items from the catalog by using the one or more hardware processors to generate, for each of the candidate items, a visual compatibility score based at least in part on a style embedding for the outfit generated by attending over style embeddings for each item in the outfit.

### Claim 18 (depends on 17)

The computer system of claim 17, wherein the means for generating a recommendation is further configured to use the one or more hardware processors to generate, for each of the candidate items, a first compatibility score by:
using an autoencoder with a graph neural network to predict pairwise similarities quantifying similarity between pairs of catalog items, the catalog items including the items in the outfit and the candidate items; and
generating the first compatibility score by averaging a set of the pairwise similarities between the candidate item and each of the items from outfit.

### Claim 19 (depends on 17)

The computer system of claim 17, wherein the means for generating a recommendation is further configured to use the one or more hardware processors to generate a second compatibility score based on a decrease in uncertainty of an outfit mixture ratio in a style basis on adding the candidate item to the outfit.

### Claim 20 (depends on 17)

The computer system of claim 17, wherein the means for generating a recommendation is further configured to use the one or more hardware processors to generate the visual compatibility score for each of the candidate items by:
generating, using a first neural network comprising an autoencoder with a graph neural network, a first compatibility score quantifying compatibility, between the outfit and the candidate item;
generating, using a second neural network configured to generate the style embedding for the outfit, a second compatibility score quantifying visual compatibility between style of the outfit and style of the candidate item; and
generating the visual compatibility score by applying a learned transformation function to combine and weight the first compatibility score and the second compatibility score.
