# US 2025/0148003 A1 · System and Method for Modifying Search Metrics Based on Features of Interest Determined from Interactions with Images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148003 A1
- Document ID: US20250148003A1
- Status: Published application
- Kind: A1
- Application number: 18/502,375
- Filed: 2023-11-06
- Published: 2025-05-08
- Earliest claimed: 2023-11-06
- Assignees: Shopify Inc.
- Applicants: Shopify Inc.
- Inventors: Rohit JACOB
- Web page: https://patentgrep.com/patent/US20250148003A1

## CPC

- G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces
- G06F 16/532: Query formulation, e.g. graphical querying
- G06F 16/583: using metadata automatically derived from the content
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system and method are provided for modifying search metrics based on features of interest determined from interactions with images. The method includes determining an input comprising at least one feature of interest, the at least one feature of interest determined from at least one interaction with a first image; and using the input in an image search by modifying a search metric to bias the image search towards locating one or more second images based on the at least one feature of interest.

## Claims

### Claim 1

A computer-implemented method comprising:
responsive to detecting at least one interaction with a first image, determining at least one feature of interest associated with the interaction; and
using the feature of interest for an image search by modifying a search metric to bias the image search towards locating one or more second images based on the at least one feature of interest.

### Claim 2 (depends on 1)

The method of claim 1, wherein the at least one feature of interest is determined from the at least one interaction with the first image by associating the at least one interaction with one or more portions of the first image.

### Claim 3 (depends on 2)

The method of claim 2, wherein the at least one feature of interest is determined by applying a feature extraction technique to at least the one or more portions of the first image.

### Claim 4 (depends on 3)

The method of claim 3, wherein the one or more portions of the first image comprise an edited portion of the first image.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
providing an image editing tool;
obtaining the at least one interaction with the first image based on edits to the first image made using the editing tool;
associating the edits with the one or more portions of the first image; and
applying the feature extraction technique to identify the at least one feature of interest according to what was edited using the editing tool.

### Claim 6 (depends on 3)

The method of claim 3, wherein the feature extraction technique applies a scale invariant feature transform (SIFT) algorithm.

### Claim 7 (depends on 1)

The method of claim 1, wherein the search metric comprises a feature weight.

### Claim 8 (depends on 1)

The method of claim 1, wherein the at least one feature of interest is determined by:
comparing an original first image to a modified first image to determine one or more changes; and
associating the one or more changes with the at least one feature of interest.

### Claim 9 (depends on 8)

The method of claim 8, wherein the comparing is performed by prompting a large language model (LLM) to describe what has changed between the original first image and the modified first image.

### Claim 10 (depends on 9)

The method of claim 9, wherein the one or more changes comprise an edited portion of the original first image.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
providing an image editing tool;
obtaining the at least one interaction with the first image based on edits to the original first image made using the editing tool to generate the modified first image;
associating the edits with one or more portions of the original first image; and
applying the feature extraction technique to identify the at least one feature of interest according to what was edited using the editing tool.

### Claim 12 (depends on 1)

The method of claim 1, further comprising training at least one model associated with one or more of: i) the at least one interaction, ii) associating the at least one interaction with one or more portions of the first image, or iii) the image search; based on an item category associated with the at least one feature of interest.

### Claim 13 (depends on 12)

The method of claim 12, wherein the item category comprises a product category.

### Claim 14 (depends on 1)

The method of claim 1, wherein the at least one interaction comprises selection of a feature, zooming of a feature, annotating a feature, or identifying a feature using text, voice or eye gaze.

### Claim 15 (depends on 1)

The method of claim 1, wherein the first image is obtained from a first search and the image search corresponds to a subsequent search.

### Claim 16

A system comprising:
at least one processor; and
at least one memory, the at least one memory comprising processor executable instructions that, when executed by the at least one processor, cause the system to:
responsive to detecting at least one interaction with a first image, determine at least one feature of interest associated with the interaction; and
use the feature of interest for an image search by modifying a search metric to bias the image search towards locating one or more second images based on the at least one feature of interest.

### Claim 17 (depends on 16)

The system of claim 16, wherein the at least one feature of interest is determined from the at least one interaction with the first image by associating the at least one interaction with one or more portions of the first image.

### Claim 18 (depends on 17)

The system of claim 17, wherein the at least one feature of interest is determined by applying a feature extraction technique to at least the one or more portions of the first image.

### Claim 19 (depends on 16)

The system of claim 16, wherein the at least one feature of interest is determined by:
comparing an original first image to a modified first image to determine one or more changes; and
associating the one or more changes with the at least one feature of interest.

### Claim 20

A non-transitory computer-readable medium comprising processor executable instructions that, when executed by a processor, cause the processor to:
responsive to detecting at least one interaction with a first image, determine at least one feature of interest associated with the interaction; and
use the feature of interest for an image search by modifying a search metric to bias the image search towards locating one or more second images based on the at least one feature of interest.
