# US 12,602,846 B2 · Generating realistic machine learning-based product images for online catalogs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,846 B2
- Document ID: US12602846B2
- Status: Granted
- Kind: B2
- Application number: 18/236,346
- Filed: 2023-08-21
- Published: 2025-02-27
- Granted: 2026-04-14
- Earliest claimed: 2023-08-21
- Assignees: Maplebear Inc.
- Applicants: Maplebear Inc.
- Inventors: Prithvishankar Srinivasan; Shih-Ting Lin; Min Xie; Shishir Kumar Prasad; Yuanzheng Zhu; Katie Ann Forbes
- Web page: https://patentgrep.com/patent/US12602846B2

## CPC

- G06F 16/583: using metadata automatically derived from the content
- G06F 16/55: Clustering; Classification
- G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 11/23: using straight lines or curves
- G06T 11/60: Creating or editing images; Combining images with text
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An online concierge system trains a fine-tuned generative image model for distinct categories of items based on a generative image model that takes a textual query as input and outputs and an associated image. Training of the fine-tuned generative image model is additionally based on a small set of representative images associated with the various categories, as well as textual tokens associated with the categories. Once trained, the fine-tuned generative image model can be used to generate realistic representative images for items in a database of the online concierge system that are lacking associated images. The fine-tuned model permits the generation of different variants of an item, such as different quantities or amounts, different packaging or packing density, and the like.

## Claims

### Claim 1

A method performed at a computer system comprising a processor and a computer-readable medium, the method for generating and managing synthetic images of items and comprising:
accessing a generative diffusion model that accepts a textual query as input and generates a synthetic image as output;
for each of a plurality of categories from a taxonomy of item categories:
generating a token corresponding to the category and not corresponding to others of the item categories;
generating a fine-tuned generative diffusion model for the categories using the generative diffusion model, sets of images corresponding to the categories, and the tokens corresponding to the categories, the fine-tuned generative diffusion model accepting a textual query as input and generating a synthetic image as output;
identifying an item in an order lacking a corresponding representative image in an database;
generating a representative image for the item using the fine-tuned generative diffusion model by:
identifying a category, from the plurality of categories arranged within a hierarchy of taxonomy, corresponding to the item in an order lacking the corresponding representative image,
identifying a token associated with that category,
generating a textual query for the category, wherein the textual query includes the token, and
specifying the textual query as input to the fine-tuned generative diffusion model;
storing the generated representative image in the database in association with the item; and
transmitting, to a picker device associated with a picker fulfilling the order through obtainment of items in the order, the order inclusive of the generated representative image for the item for display of the generated representative image on the picker device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving a query of a customer for items from the database;
identifying items of the database corresponding to the query, the items including the item for which the representative image was generated; and
causing display of data about the items, the data including the representative image in visual association with the item.

### Claim 3 (depends on 1)

The method of claim 1, wherein the textual query contains the token corresponding to the category of the item lacking the corresponding representative image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the textual query contains at least one of: an amount of the item, a quantity of the item, a packaging density of the item, or a packaging type of the item.

### Claim 5 (depends on 1)

The method of claim 1, wherein the token is randomly generated.

### Claim 6 (depends on 1)

The method of claim 1, wherein the fine-tuned generative diffusion model is generated using DREAMBOOTH.

### Claim 7

A non-transitory computer-readable storage medium containing instructions that when executed by one or more processors perform actions comprising:
accessing a generative diffusion model that accepts a textual query as input and generates a synthetic image as output;
for each of a plurality of categories from a taxonomy of item categories:
generating a token corresponding to the category and not corresponding to any of the other item categories;
generating a fine-tuned generative diffusion model for the categories using the generative diffusion model, sets of images corresponding to the categories, and the tokens corresponding to the categories, the fine-tuned generative diffusion model accepting a textual query as input and generating a synthetic image as output;
identifying an item lacking a corresponding representative image in a database;
generating a representative image for the item using the fine-tuned generative diffusion model by:
identifying a category, from the plurality of categories arranged within a hierarchy of taxonomy, corresponding to the item in an order lacking the corresponding representative image,
identifying a token associated with that category,
generating a textual query for the category, wherein the textual query includes the token, and
specifying the textual query as input to the fine-tuned generative diffusion model;
storing the generated representative image in the database in association with the item; and
transmitting, to a picker device associated with a picker fulfilling the order through obtainment of items in the order, the order inclusive of the generated representative image for the item for display of the generated representative image on the picker device.

### Claim 8 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, the actions further comprising:
receiving a query of a customer for items from the database;
identifying items of the database corresponding to the query, the items including the item for which the representative image was generated; and
causing display of data about the items, the data including the representative image in visual association with the item.

### Claim 9 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the textual query contains the token corresponding to the category of the item lacking the corresponding representative image.

### Claim 10 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the textual query contains at least one of: an amount of the item, a quantity of the item, a packaging density of the item, or a packaging type of the item.

### Claim 11 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the token is randomly generated.

### Claim 12 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the fine-tuned generative diffusion model is generated using DREAMBOOTH.

### Claim 13

A computer system comprising:
one or more computer processors; and
a computer-readable storage medium storing instructions that when executed by the one or more computer processors perform actions comprising:
accessing a generative diffusion model that accepts a textual query as input and generates a synthetic image as output;
for each of a plurality of categories from a taxonomy of item categories:
generating a token corresponding to the category and not corresponding to any of the other item categories;
generating a fine-tuned generative diffusion model for the categories using the generative diffusion model, sets of images corresponding to the categories, and the tokens corresponding to the categories, the fine-tuned generative diffusion model accepting a textual query as input and generating a synthetic image as output;
identifying an item lacking a corresponding representative image in a database;
generating a representative image for the item using the fine-tuned generative diffusion model by:
identifying a category, from the plurality of categories arranged within a hierarchy of taxonomy, corresponding to the item in an order lacking the corresponding representative image,
identifying a token associated with that category,
generating a textual query for the category, wherein the textual query includes the token, and
specifying the textual query as input to the fine-tuned generative diffusion model;
storing the generated representative image in the database in association with the item; and
transmitting, to a picker device associated with a picker fulfilling the order through obtainment of items in the order, the order inclusive of the generated representative image for them for display of the generated representative image on the picker device.

### Claim 14 (depends on 13)

The computer system of claim 13, the actions further comprising:
receiving a query of a customer for items from the database;
identifying items of the database corresponding to the query, the items including the item for which the representative image was generated; and
causing display of data about the items, the data including the representative image in visual association with the item.

### Claim 15 (depends on 13)

The computer system of claim 13, wherein the textual query contains the token corresponding to the category of the item lacking the corresponding representative image.

### Claim 16 (depends on 13)

The computer system of claim 13, wherein the textual query contains at least one of: an amount of the item, a quantity of the item, a packaging density of the item, or a packaging type of the item.

### Claim 17 (depends on 13)

The computer system of claim 13, wherein the fine-tuned diffusion model is generated using DREAMBOOTH.