# US 12,118,046 B2 · Systems and methods for modeling item similarity using converted image information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,046 B2
- Document ID: US12118046B2
- Status: Granted
- Kind: B2
- Application number: 17/653,151
- Filed: 2022-03-02
- Published: 2022-07-28
- Granted: 2024-10-15
- Earliest claimed: 2021-01-25
- Assignees: Coupang Corp.
- Applicants: Coupang Corp.
- Inventors: Nuri Mehmet Gokhan; Varun A Samtani
- Web page: https://patentgrep.com/patent/US12118046B2

## CPC

- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06F 16/951: Indexing; Web crawling techniques
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/23: Clustering techniques
- G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification}
- G06N 3/02: Neural networks
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- G06Q 30/0283: Price estimation or determination
- G06T 7/194: involving foreground-background segmentation
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/40: Extraction of image or video features
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G10L 17/10: Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems
- G06V 2201/10: Recognition assisted with metadata
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods for correlating item data are disclosed. A system for correlating item data may include a memory storing instructions and at least one processor configured to execute instructions to perform operations including: receiving text and image data associated with a reference item from a remote device; converting, using a computer-modeled embedding layer, at least one image to an image embedding; comparing the image embedding to reference embeddings stored in a database; selecting a subset of the candidate item text as candidate text data based on the comparison; selecting a subset of the candidate item images as candidate image data based on the comparison; selecting a text correlation model; determining a first similarity score; selecting an image correlation model; determining a second similarity score; calculating a confidence score based on the first and second similarity scores; and performing a responsive action based on the calculated confidence score.

## Claims

### Claim 1

A system for correlating item data, the system comprising:
at least one processor; and
a non-transitory computer-readable medium containing a set of instructions that, when executed by the at least one processor, cause the processor to perform steps comprising:
receiving text data associated with a reference item;
receiving image data comprising at least one image associated with the reference item;
converting, using a computer-modeled embedding layer, the at least one image to an image embedding;
comparing the image embedding to reference embeddings, the stored reference embeddings being associated with candidate item images and candidate item text of candidate items;
selecting, based on the comparison, candidate text data, wherein the candidate text data is a subset of the candidate item text and includes a price;
selecting, based on the comparison, candidate image data, the candidate image data being a subset of the candidate item images;
selecting a text correlation model;
determining a first similarity score by applying the selected text correlation model to the received text data and the subset of the candidate item text;
selecting an image correlation model;
determining a second similarity score by applying the selected image correlation model to the received image data and the subset of the candidate item images;
calculating a confidence score based on the determined first and second similarity scores; and
performing a responsive action based on the calculated confidence score, wherein the responsive action comprises at least one of:
creating an association between the reference item and one of the candidate items;
monitoring a webpage associated with the reference item;
updating at least one of the reference embeddings;
modifying the selected text correlation model; or
modifying the selected image correlation model.

### Claim 2 (depends on 1)

The system of claim 1, wherein the stored reference embeddings are clustered in a Euclidean space.

### Claim 3 (depends on 1)

The system of claim 1, wherein the responsive action comprises updating at least one of the reference embeddings by performing at least one of:
adding a reference embedding to the stored reference embeddings;
removing a reference embedding from the stored reference embeddings; or
adjusting a Euclidean-space boundary associated with the stored reference embeddings.

### Claim 4 (depends on 1)

The system of claim 1, wherein the received text data is crawled from a web page by a text crawler configured to search for item text data associated with a particular item.

### Claim 5 (depends on 1)

The system of claim 1, wherein the received image data is crawled from a web page by an image crawler configured to search for item image data associated with a particular item.

### Claim 6 (depends on 1)

The system of claim 1, wherein the candidate item text is tokenized prior to the selection of the candidate text data.

### Claim 7 (depends on 1)

The system of claim 1, wherein the received text data is tokenized prior to applying the selected text correlation model to the received text data and the subset of the candidate item text.

### Claim 8 (depends on 1)

The system of claim 1, wherein at least one of the text correlation model or the image correlation model is selected based on a category of the reference item.

### Claim 9 (depends on 1)

The system of claim 1, wherein:
at least one of the received text data or the received image data is crawled from a web page by a crawler; and
at least one of the selected text correlation model or the selected image correlation model is trained using data sourced from a host of the web page.

### Claim 10 (depends on 1)

The system of claim 1, wherein at least one of the selected text correlation model or the selected image correlation model is a random forest model.

### Claim 11 (depends on 1)

The system of claim 1, wherein the selected text correlation model contains a text frequency parameter having a weight that is inversely related to a frequency of a character combination in a reference dataset.

### Claim 12 (depends on 1)

The system of claim 1, wherein the selected image correlation model is configured to segment the at least one image into segments and generate a unique identifier for one of the segments of the at least one image.

### Claim 13 (depends on 12)

The system of claim 12, wherein the selected image correlation model is configured to compare the generated unique identifier to an identifier generated for a segment of at least one of the candidate item images.

### Claim 14 (depends on 1)

The system of claim 1, wherein the selected image correlation model is configured to ignore an image property identified by a user input.

### Claim 15 (depends on 1)

The system of claim 1, wherein the calculated confidence score is a statistical combination of the determined first and second similarity scores.

### Claim 16 (depends on 1)

The system of claim 1, wherein the calculated confidence score is based on a previous match between the reference item and a candidate item.

### Claim 17

A computer-implemented method for correlating item data, the method comprising:
receiving text data associated with a reference item;
receiving image data comprising at least one image associated with the reference item;
converting, using a computer-modeled embedding layer, the at least one image to an image embedding;
comparing the image embedding to reference embeddings, the stored reference embeddings being associated with candidate item images and candidate item text of candidate items;
selecting, based on the comparison, candidate text data, wherein the candidate text data is a subset of the candidate item text and includes a price;
selecting, based on the comparison, candidate image data, the candidate image data being a subset of the candidate item images;
selecting a text correlation model;
determining a first similarity score by applying the selected text correlation model to the received text data and the subset of the candidate item text;
selecting an image correlation model;
determining a second similarity score by applying the selected image correlation model to the received image data and the subset of the candidate item images;
calculating a confidence score based on the determined first and second similarity scores; and
performing a responsive action based on the calculated confidence score, wherein the responsive action comprises at least one of:
creating an association between the reference item and one of the candidate items;
monitoring a webpage associated with the reference item;
updating at least one of the reference embeddings;
modifying the selected text correlation model; or
modifying the selected image correlation model.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein the received text data is crawled from a web page by a text crawler configured to search for item text data associated with a particular item.

### Claim 19 (depends on 17)

The computer-implemented method of claim 17, wherein the received image data is crawled from a web page by an image crawler configured to search for item image data associated with a particular item.

### Claim 20 (depends on 17)

The computer-implemented method of claim 17, wherein the selected image correlation model is configured to segment the at least one image into segments and generate a unique identifier for one of the segments of the at least one image.