# US 11,907,338 B2 · Retrieving images that correspond to a target subject matter within a target context

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,338 B2
- Document ID: US11907338B2
- Status: Granted
- Kind: B2
- Application number: 17/158,170
- Filed: 2021-01-26
- Published: 2022-07-28
- Granted: 2024-02-20
- Earliest claimed: 2021-01-26
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Gourav Singhal; Sourabh Gupta; Mrinal Kumar Sharma
- Web page: https://patentgrep.com/patent/US11907338B2

## CPC

- G06F 18/24: Classification techniques
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/40: Extraction of image or video features
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06V 10/235: {based on user input or interaction}
- G09G 2354/00: Aspects of interface with display user
- G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally
- G09G 2380/04: Electronic labels
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Techniques are provided herein for retrieving images that correspond to a target subject matter within a target context. Although useful in a number of applications, the techniques provided herein are particularly useful in contextual product association and visualization. A method is provided to apply product images to a neural network. The neural network is configured to classify the products in the images. The images are associated with a context representing the combination of classified products in the images. These techniques leverage both seller-provided images of products and user-generated content, which potentially includes hundreds or thousands of images of the same or similar products as the seller-provided images. A graphical user interface is configured to permit a user to select the context of interest in which to visualize the products.

## Claims

### Claim 1

A method for determining a context of a main product and at least one context product in an image, the method comprising:
classifying, by a product identification module, a first context product in a feature map using a first neural network having one or more product classification layers, the feature map representing features in a first image containing the main product and in a second image containing the main product;
classifying, by the product identification module, a second context product in the feature map using the first neural network;
classifying, by a prominent region detection module, a common prominent feature in the feature map using a second neural network having one or more prominent feature classification layers, the common prominent feature being different from the first context product and the second context product; and
associating, by a product association module and responsive to classifying the common prominent feature, the first image with the first context product and further associating the second image with the second context product.

### Claim 2 (depends on 1)

The method of claim 1, further comprising causing, by a product display module, the first image to be displayed on a display device in response to a user selection of the first context product, and the second image to be displayed on the display device in response to a user selection of the second context product.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the product identification module and from the first neural network, a bounding box surrounding the first context product in the first image;
generating, by the product identification module, a mask corresponding to the first context product within the bounding box;
assigning, by the prominent region detection module, a weight to the mask; and
identifying, by the prominent region detection module and from the second neural network, the common prominent feature based on the mask and the weight.

### Claim 4 (depends on 3)

The method of claim 3, wherein the weight is a first weight, wherein the method further comprises assigning, by the prominent region detection module, a second weight to a region of the first image containing the main product, and wherein identifying the common prominent feature is further based on a combination of the first and second weights.

### Claim 5 (depends on 1)

The method of claim 1, further comprising obtaining, by the product identification module and from the first neural network, a first label associated with the main product, a second label associated with the first context product, and a third label associated with the second context product, wherein the first image is associated with the first context product based on the first and second labels, and wherein the second image is associated with the second context product based on the first and third labels.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first neural network is a convolutional neural network (CNN) trained on a product image training dataset, the product image training dataset including the first label, the second label, and the third label.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first context product is the same as the second context product, and wherein the method further comprises:
applying, by the product display module, the first image and the second image to a third neural network configured to generate an image quality score for each of the first image and the second image; and
generating, by the product display module, a thumbnail image from one of the first image and the second image having the highest image quality score.

### Claim 8 (depends on 7)

The method of claim 7, further comprising causing, by the product display module, the thumbnail image to be displayed on the display device in response to a user selection of the main product.

### Claim 9 (depends on 1)

The method of claim 1, wherein there are a plurality of first images containing the main product, and wherein the method further comprises:
associating, by the product association module and responsive to identifying the common prominent feature, each of the plurality of first images with the first context product; and
causing, by the product display module, each of the plurality of first images to be displayed on the display device in response to a user selection of the first context product.

### Claim 10

A system for determining a context of a main product and at least one context product in an image, the system comprising:
a product identification module, executable by at least one processor, and configured to
classify, using a first neural network having one or more product classification layers, a first context product in a feature map, the feature map representing features in a first image containing the main product and in a second image containing the main product, and
classify, using the first neural network, a second context product in the feature map;
a prominent region detection module, executable by the at least one processor, and configured to classify, using a second neural network, a common prominent feature in the feature map, the common prominent feature being different from the first context product and the second context product;
a product association module, executable by the at least one processor responsive to classifying the common prominent feature, and configured to
associate the first image with the first context product, and
associate the second image with the second context product; and
a product display module, executable by the at least one processor, and configured to receive a user selection of the first context product or the second context product, and
cause the first image to be displayed via a display device in response to the user selection of the first context product, and the second image to be displayed via the display device in response to the user selection of the second context product.

### Claim 11 (depends on 10)

The system of claim 10, wherein the product identification module is further configured to:
obtain, from the first neural network, a bounding box surrounding the first context product in the first image;
wherein the product identification module is further configured to generate a mask corresponding to the first context product within the bounding box; and
wherein the prominent region detection module is further configured to
assign a weight to the mask, and
identify, using the second neural network, the common prominent feature based on the mask and the weight.

### Claim 12 (depends on 10)

The system of claim 10, wherein the product identification module is further configured to obtain, from the first neural network, a first label associated with the main product, a second label associated with the first context product, and a third label associated with the second context product, wherein the first image is associated with the first context product based on the first and second labels, and wherein the second image is associated with the second context product based on the first and third labels.

### Claim 13 (depends on 10)

The system of claim 10, wherein the first neural network is a convolutional neural network (CNN) trained on a product image training dataset, the product image training dataset including the first label, the second label, and the third label.

### Claim 14 (depends on 10)

The system of claim 10, wherein the first context product is the same as the second context product, and wherein the product display module is further configured to:
apply the first image and the second image to a third neural network configured to generate an image quality score for each of the first image and the second image;
generate a thumbnail image from one of the first image and the second image having the highest image quality score; and
cause the thumbnail image to be displayed on the display device in response to a user selection of the main product.

### Claim 15

A computer program product including one or more non-transitory machine-readable mediums having instructions encoded thereon that when executed by at least one processor cause a process to be carried out for determining a context of a main product and at least one context product in an image, the process comprising:
classifying a first context product in a feature map using a first neural network having one or more product classification layers, the feature map representing features in a first image containing the main product and in a second image containing the main product;
classifying a second context product in the feature map using the first neural network;
classifying a common prominent feature in the feature map using a second neural network having one or more prominent feature classification layers, the common prominent feature being different from the first context product and the second context product;
associating, in response to identifying the common prominent feature, the first image with the first context product and further associating the second image with the second context product; and
causing the first image to be displayed on a display device in response to a user selection of the first context product, and the second image to be displayed on the display device in response to a user selection of the second context product.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the common prominent feature in both the first image and the second image includes a region of a person's body.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the process further comprises:
obtaining, from the first neural network, a bounding box surrounding the first context product in the first image;
generating a mask corresponding to the first context product within the bounding box;
assigning a weight to the mask; and
identifying, from the second neural network, the common prominent feature based on the mask and the weight.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the weight is a first weight, wherein the process further comprises assigning a second weight to a region of the first image containing the main product, and wherein identifying the common prominent feature is further based on a combination of the first and second weights.

### Claim 19 (depends on 15)

The computer program product of claim 15, wherein the process further comprises obtaining, by the product identification module and from the first neural network, a first label associated with the main product, a second label associated with the first context product, and a third label associated with the second context product, wherein the first image is associated with the first context product based on the first and second labels, and wherein the second image is associated with the second context product based on the first and third labels.

### Claim 20 (depends on 15)

The computer program product of claim 15, wherein the first context product is the same as the second context product, and wherein the process further comprises:
applying the first image and the second image to a third neural network configured to generate an image quality score for each of the first image and the second image;
generating a thumbnail image from one of the first image and the second image having the highest image quality score; and
causing the thumbnail image to be displayed on the display device in response to a user selection of the main product.