# US 12,596,766 B2 · Automatically generating an image dataset based on object instance similarity

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,596,766 B2
- Document ID: US12596766B2
- Status: Granted
- Kind: B2
- Application number: 17/337,194
- Filed: 2021-06-02
- Published: 2022-12-08
- Granted: 2026-04-07
- Earliest claimed: 2021-06-02
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Midhun Harikumar; Zhe Lin; Shabnam Ghadar; Baldo Faieta
- Web page: https://patentgrep.com/patent/US12596766B2

## CPC

- G06F 18/211: Selection of the most significant subset of features
- G06F 18/23: Clustering techniques
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06F 18/25: Fusion techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06V 10/273: {removing elements interfering with the pattern to be recognised}
- G06V 10/82: using neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and non-transitory computer readable media are disclosed for accurately and efficiently generating groups of images portraying semantically similar objects for utilization in building machine learning models. In particular, the disclosed system utilizes metadata and spatial statistics to extract semantically similar objects from a repository of digital images. In some embodiments, the disclosed system generates color embeddings and content embeddings for the identified objects. The disclosed system can further group similar objects together within a query space by utilizing a clustering algorithm to create object clusters and then refining and combining the object clusters within the query space. In some embodiments, the disclosed system utilizes one or more of the object clusters to build a machine learning model.

## Claims

### Claim 1

A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:
select a series of correlated images from digital images utilizing metadata associated with the digital images;
extract objects portrayed in the series of correlated images by:
detecting the objects in the series of correlated images by utilizing an object detection model;
generating segmentation masks of the objects in the series of correlated images by utilizing a segmentation model; and
removing image data not associated with the segmentation masks from the series of correlated images to generate extracted objects;
generate content embeddings and color embeddings for the extracted objects;
generate object clusters by grouping semantically similar objects from the content embeddings and the color embeddings utilizing a clustering algorithm;
deduplicate the object clusters by merging nearest object clusters of the object clusters; and
utilize one or more of the deduplicated object clusters and corresponding digital images to build a machine learning model by:
selecting a pair of ground truth similar images from an object cluster of the deduplicated object clusters; and
utilizing the pair of ground truth similar images to train a digital image similarity machine learning model.

### Claim 2 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to select the series of correlated images by:
determining, based on the metadata associated with the digital images, a subset of digital images having at least one of time metadata tags within a threshold time period, location metadata tags within a threshold distance, or matching user identification metadata tags; and
utilizing the subset of digital images to select the series of correlated images.

### Claim 3 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to select the digital images by:
determining one or more historical image queries or upload keywords;
determining a set of images corresponding to the one or more historical image queries or the upload keywords; and
utilizing the set of images, extracting the digital images utilized to select the series of correlated images.

### Claim 4 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to process a plurality of digital images utilizing the machine learning model to generate a plurality of digital embeddings for use in responding to digital image queries.

### Claim 5 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the content embeddings by:
generating feature vectors from the extracted objects by utilizing a convolutional neural network trained to generate semantic image labels; and
utilizing the feature vectors as the content embeddings to generate the object clusters.

### Claim 6 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the color embeddings by:
grouping pixels of the extracted objects into a plurality of colors to generate color histograms; and
utilizing the color histograms as the color embeddings to generate the object clusters.

### Claim 7 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the object clusters by:
combining the content embeddings and the color embeddings for the extracted objects; and
mapping the combined embeddings corresponding to the extracted objects in a query space.

### Claim 8 (depends on 7)

The non-transitory computer readable medium as recited in claim 7, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the object clusters comprises utilizing a density-based clustering algorithm to generate the object clusters from the combined embeddings corresponding to the extracted objects in the query space.

### Claim 9 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to deduplicate the object clusters by:
generating a mean cluster embedding for an object cluster of the object clusters;
generating a nearest mean cluster embedding for a nearest object cluster; and
merging the nearest object cluster with the object cluster based on a distance between the nearest mean cluster embedding and the mean cluster embedding falling within a threshold distance value.

### Claim 10 (depends on 1)

The non-transitory computer readable medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to utilize the one or more of the object clusters to build the machine learning model by utilizing groups of digital images corresponding with the one or more of the object clusters as a ground truth similarity data set for building the digital image similarity machine learning model.

### Claim 11

A system comprising:
at least one memory device comprising digital images having metadata; and
at least one server device configured to cause the system to:
select a series of correlated images from the digital images utilizing the metadata associated with the digital images by grouping digital images having related metadata tags;
extract objects portrayed in the series of correlated images by generating background masks for the series of correlated images utilizing an object detection model and a segmentation model;
generate content embeddings and color embeddings for the extracted objects by utilizing a convolutional neural network trained to generate semantic image labels and a color embedding model;
generate object clusters by grouping semantically similar objects from the content embeddings and the color embeddings utilizing a clustering algorithm by mapping combined embeddings in a query space; and
deduplicate the object clusters by merging nearest object clusters of the object clusters based on a threshold distance value; and
build a digital image similarity machine learning model by:
selecting a pair of ground truth similar images from an object cluster of the deduplicated object clusters; and
utilizing the pair of ground truth similar images to train the digital image similarity machine learning model.

### Claim 12 (depends on 11)

The system as recited in claim 11, wherein the at least one server device is further configured to cause the system to generate the extracted objects by removing image data not associated with the background masks from the series of correlated images.

### Claim 13 (depends on 11)

The system as recited in claim 11, wherein the at least one server device is further configured to cause the system to utilize the digital image similarity machine learning model by processing a plurality of digital images utilizing the digital image similarity machine learning model to generate a plurality of digital image embeddings for use in responding to digital image queries.

### Claim 14 (depends on 11)

The system as recited in claim 11, wherein the at least one server device is further configured to cause the system to:
utilize one or more of the deduplicated object clusters and instance labels associated with the one or more of the deduplicated object clusters to build an object classification machine learning model; and
utilize the object classification machine learning model to generate predicted instance labels for objects portrayed in a plurality of images.

### Claim 15 (depends on 11)

The system as recited in claim 11, wherein the at least one server device is further configured to generate the background masks for the series of correlated images by:
detecting the objects in the series of correlated images by utilizing the object detection model;
generating segmentation masks of the objects in the series of correlated images by utilizing the segmentation model; and
generating the background masks by inverting the segmentation masks.

### Claim 16

In a digital medium environment for processing a repository of digital images, a computer-implemented method comprising:
select a series of correlated images from digital images utilizing metadata associated with the digital images;
extracting objects portrayed in a series of correlated images by:
detecting the objects in the series of correlated images by utilizing an object detection model;
generating segmentation masks of the objects in the series of correlated images by utilizing a segmentation model; and
removing image data not associated with the segmentation masks from the series of correlated images to generate extracted objects;
generating content embeddings and color embeddings for the extracted objects;
generating object clusters by grouping the content embeddings and the color embeddings utilizing a clustering algorithm;
utilizing one or more of the object clusters and corresponding digital images to build a digital image similarity machine learning model;
deduplicating the object clusters by merging nearest object clusters of the object clusters based on a threshold distance value; and
utilizing one or more of the deduplicated object clusters to build the digital image similarity machine learning model by:
selecting a pair of ground truth similar images from an object cluster of the deduplicated object clusters; and
utilizing the pair of ground truth images to train the digital image similarity machine learning model.

### Claim 17 (depends on 16)

The computer-implemented method as recited in claim 16, further comprising generating the content embeddings by:
generating feature vectors from the extracted objects by utilizing a convolutional neural network trained to generate semantic image labels; and
utilizing the feature vectors as the content embeddings to generate the object clusters.

### Claim 18 (depends on 16)

The computer-implemented method as recited in claim 16, further comprising generating the color embeddings by generating color histograms from the extracted objects.

### Claim 19 (depends on 16)

The computer-implemented method as recited in claim 16, further comprising:
processing a plurality of digital images utilizing the digital image similarity machine learning model to generate a plurality of digital image embeddings for use in responding to digital image queries.

### Claim 20 (depends on 16)

The computer-implemented method as recited in claim 16, further comprising generating the object clusters by:
combining the content embeddings and the color embeddings for the extracted objects; and
mapping the combined embeddings corresponding to the extracted objects in a query space.