# US 12,183,058 B2 · Systems and methods for training and using a machine learning model for matching objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,058 B2
- Document ID: US12183058B2
- Status: Granted
- Kind: B2
- Application number: 17/672,755
- Filed: 2022-02-16
- Published: 2023-08-17
- Granted: 2024-12-31
- Earliest claimed: 2022-02-16
- Assignees: Shopify Inc.
- Applicants: Shopify Inc.
- Inventors: Shaked Dunay; Adam Malloul; Roni Gurvich
- Web page: https://patentgrep.com/patent/US12183058B2

## CPC

- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/764: using classification, e.g. of video objects
- G06F 16/532: Query formulation, e.g. graphical querying
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06Q 30/0621: {by configuring or customising goods or services}
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0631: {Recommending goods or services}
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/443: {by matching or filtering}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer application may aim to identify first and second “matching” objects. The matching method cannot necessarily be based on how visually similar the two objects are to each other because two matching objects might be different and/or be visually different. Moreover, the images of the objects to be matched might not necessarily have metadata to assist in the matching. In some embodiments, a machine learning model may be trained using a set of digital images, each including two or more matching objects. Triplet loss training may be used, and each triplet may include: an image of a first object extracted from a first image, an image of an object that is visually similar to an image of a second object extracted from the first image, and an image of a third object extracted from a different image.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining a first image of a first object extracted from an original image;
obtaining a second image of a second object extracted from the original image;
using the second image to obtain a similar image, wherein the similar image is visually similar to the second image and is not obtained from the original image; and
training a machine learning model using a triplet including:
the first image of the first object,
the similar image, and
a third image of a third object that is not extracted from the original image.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the similar image is at least one of:
an image obtained using a trained similarity machine learning model,
an alternative image of the second object, or
an image created by modifying the second image of the second object.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein obtaining the image using the trained similarity machine learning model comprises inputting the second image into the trained similarity machine learning model, and wherein the trained similarity machine learning model is trained to:
obtain a vector representation of the second image,
identify a corresponding image having a vector representation that is within a predetermined distance from the vector representation of the second image, and
output an indication of the corresponding image, the corresponding image being the similar image.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein extracting the first object and the second object from the original image comprises:
identifying the first object and the second object in the original image;
isolating a first region of the original image including the first object and a first contextual element;
isolating a second region of the original image including the second object and a second contextual element; and
storing the first region as the first image and the second region as the second image.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein:
the first image is an anchor of the triplet, the similar image is a positive of the triplet, and the third image is a negative of the triplet,
the training including inputting each of the anchor, the positive, and the negative into the machine learning model to obtain vector representations of the anchor, the positive, and the negative, and computing a loss value based on a difference between a first distance and a second distance, the first distance based on a distance between the vector representations of the anchor and positive, and the second distance based on a distance between the vector representations of the anchor and negative.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the training uses a set of triplets that includes a subset of triplets for the original image, wherein for each triplet of the subset of triplets:
an anchor of the triplet is the first image, and one of:
a positive of the triplet is the similar image and a negative of the triplet is another image different from the third image and that is not extracted from the original image, or
the positive of the triplet is another image different from the similar image and that is visually similar to the second image and the negative of the triplet is the third image, or
the positive of the triplet is another image that is visually similar to the second image and the negative of the triplet is another image that is not extracted from the original image.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein, after training the machine learning model to obtain a trained machine learning model, the method further comprises:
receiving, from a device associated with a user, a request for web content, wherein the web content includes an image of a first product;
determining one or more matching products based on distances between a vector representation of the image of the first product and vector representations of other products obtained using the trained machine learning model; and
transmitting, for display on the user device, the web content including one or more images of the one or more matching products.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein, if a vector representation of the image of the first product is not already stored, the method further comprises inputting the image of the first product into the trained machine learning model to obtain the vector representation of the image of the first product.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the third image of the third object is extracted from a different image having a same source as the original image, wherein the source is a set of digital images that originate from a same image collection.

### Claim 10

A system comprising:
a memory to store images including an original image; and
at least one processor to:
obtain a first image of a first object extracted from the original image;
obtain a second image of a second object extracted from the original image;
use the second image to obtain a similar image, wherein the similar image is visually similar to the second image and is not obtained from the original image; and
train a machine learning model using a triplet including:
the first image of the first object,
the similar image, and
a third image of a third object that is not extracted from the original image.

### Claim 11 (depends on 10)

The system of claim 10, wherein the similar image is at least one of:
an image obtained using a trained similarity machine learning model,
an alternative image of the second object, or
an image created by modifying the second image of the second object.

### Claim 12 (depends on 11)

The system of claim 11, wherein the at least one processor is to obtain the image using the trained similarity machine learning model by performing operations including inputting the second image into the trained similarity machine learning model, and wherein the trained similarity machine learning model is trained to:
obtain a vector representation of the second image,
identify a corresponding image having a vector representation that is within a predetermined distance from the vector representation of the second image, and
output an indication of the corresponding image, the corresponding image being the similar image.

### Claim 13 (depends on 10)

The system of claim 10, wherein to extract the first object and the second object from the original image, the at least one processor is to:
identify the first object and the second object in the original image;
isolate a first region of the original image including the first object and a first contextual element;
isolate a second region of the original image including the second object and a second contextual element; and
store the first region as the first image and the second region as the second image.

### Claim 14 (depends on 10)

The system of claim 10, wherein:
the first image is an anchor of the triplet, the similar image is a positive of the triplet, and the third image is a negative of the triplet,
the at least one processor is to train the machine learning model by performing operations including inputting each of the anchor, the positive, and the negative into the machine learning model to obtain vector representations of the anchor, the positive, and the negative,
and the at least one processor is to compute a loss value based on a difference between a first distance and a second distance, the first distance based on a distance between the vector representations of the anchor and positive, and the second distance based on a distance between the vector representations of the anchor and negative.

### Claim 15 (depends on 10)

The system of claim 10, wherein training the machine learning model uses a set of triplets that includes a subset of triplets for the original image, wherein for each triplet of the subset of triplets:
an anchor of the triplet is the first image, and one of:
a positive of the triplet is the similar image and a negative of the triplet is another image different from the third image and that is not extracted from the original image, or
the positive of the triplet is another image different from the similar image and that is visually similar to the second image and the negative of the triplet is the third image, or
the positive of the triplet is another image that is visually similar to the second image and the negative of the triplet is another image that is not extracted from the original image.

### Claim 16 (depends on 10)

The system of claim 10, wherein, after training the machine learning model to obtain a trained machine learning model, the at least one processor is to:
receive, from a device associated with a user, a request for web content, wherein the web content includes an image of a first product;
determine one or more matching products based on distances between a vector representation of the image of the first product and vector representations of other products obtained using the trained machine learning model; and
instruct transmission of the web content for display on the user device, the web content including one or more images of the one or more matching products.

### Claim 17 (depends on 16)

The system of claim 16, wherein, if a vector representation of the image of the first product is not already stored, the at least one processor is to input the image of the first product into the trained machine learning model to obtain the vector representation of the image of the first product.

### Claim 18 (depends on 10)

The system of claim 10, wherein the at least one processor is to extract the third image of the third object from a different image having a same source as the original image, wherein the source is a set of digital images that originate from a same image collection.

### Claim 19

A non-transitory computer readable medium having stored thereon computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
obtaining a first image of a first object extracted from an original image;
obtaining a second image of a second object extracted from the original image;
using the second image to obtain a similar image, wherein the similar image is visually similar to the second image and is not obtained from the original image; and
training a machine learning model using a triplet including:
the first image of the first object,
the similar image, and
a third image of a third object that is not extracted from the original image.