# US 2024/0256793 A1 · Methods and systems for generating text with tone or diction corresponding to stylistic attributes of images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256793 A1
- Document ID: US20240256793A1
- Status: Published application
- Kind: A1
- Application number: 18/180,566
- Filed: 2023-03-08
- Published: 2024-08-01
- Earliest claimed: 2023-01-31
- Applicants: Shopify Inc.
- Inventors: Russ MASCHMEYER
- Web page: https://patentgrep.com/patent/US20240256793A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- G06V 10/40: Extraction of image or video features
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for prompting a large language model (LLM) to generate a stylistic description of an image are disclosed. One or more visual attributes are extracted from an image using a first trained machine learning model. The visual attributes are mapped to one or more emotion attributes using a second trained machine learning model. A LLM prompt is generated based on the one or more emotion attributes and provided to the LLM. A generated description of the image is obtained from the LLM and displayed with the image.

## Claims

### Claim 1

A system comprising:
a processing unit configured to execute instructions to cause the system to:
extract, from an image, one or more visual attributes of the image using a first trained machine learning model;
map the one or more visual attributes to one or more emotion attributes using a second trained machine learning model;
generate a prompt to a large language model (LLM), the prompt being based on the one or more emotion attributes;
provide the generated prompt to the LLM; and
obtain, from the LLM, a generated description of the image.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first trained machine learning model is a trained deep neural network.

### Claim 3 (depends on 2)

The system of claim 2, wherein the second trained machine learning model is a trained neural network.

### Claim 4 (depends on 1)

The system of claim 1, wherein the prompt includes at least one of the one or more emotion attributes.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processing unit is further configured to execute instructions to cause the system to incorporate a generic description of the image into the prompt.

### Claim 6 (depends on 5)

The system of claim 5, wherein the processing unit is further configured to execute instructions to cause the system to retrieve the generic description of the object from a description database.

### Claim 7 (depends on 5)

The system of claim 5, wherein the processing unit is further configured to execute instructions to cause the system to provide the image to a descriptor text generator to obtain the generic description for incorporation into the prompt.

### Claim 8 (depends on 1)

The system of claim 1, wherein the image comprises an object.

### Claim 9 (depends on 8)

The system of claim 8, wherein the generated prompt further comprises a name of the object in the image.

### Claim 10 (depends on 8)

The system of claim 8, wherein the processing unit is further configured to execute instructions to cause the system to incorporate physical attributes of the object into the prompt.

### Claim 11 (depends on 8)

The system of claim 8, wherein the visual attributes are extracted from multiple images of the object.

### Claim 12 (depends on 11)

The system of claim 11, wherein the visual attributes are visual attributes that were common to each of the multiple images.

### Claim 13

A computer-implemented method comprising:
extracting, from an image, one or more visual attributes of the image using a first trained machine learning model;
mapping the one or more visual attributes to one or more emotion attributes using a second trained machine learning model;
generating a prompt to a large language model (LLM), the prompt being based on the one or more emotion attributes;
providing the generated prompt to the LLM; and
obtaining, from the LLM, a generated description of the image.

### Claim 14 (depends on 13)

The method of claim 13, wherein the first trained machine learning model is a trained deep neural network.

### Claim 15 (depends on 14)

The system of claim 14, wherein the second trained machine learning model is a trained neural network.

### Claim 16 (depends on 15)

The method of claim 15, wherein generating the prompt comprises incorporating at least one of the one or more emotion attributes into the prompt.

### Claim 17 (depends on 16)

The method of claim 16, wherein generating the prompt comprises incorporating a generic description of the image into the prompt.

### Claim 18 (depends on 17)

The method of claim 17, wherein generating the prompt further comprises: retrieving the generic description of the object from a description database.

### Claim 19 (depends on 17)

The method of claim 17, wherein generating the prompt further comprises: providing the image to a descriptor text generator to obtain the generic description.

### Claim 20 (depends on 15)

The method of claim 15, wherein the image comprises an object.

### Claim 21 (depends on 20)

The method of claim 20, wherein generating the prompt comprises incorporating physical attributes of the object into the prompt.

### Claim 22 (depends on 21)

The method of claim 21, further comprising: extracting the physical attributes of the object from an object attribute database.

### Claim 23 (depends on 20)

The method of claim 20, wherein the visual attributes are extracted from multiple images of the object.

### Claim 24 (depends on 23)

The method of claim 23, wherein the visual attributes are visual attributes that were the most commonly extracted from the multiple images.

### Claim 25

A non-transitory computer-readable medium storing instructions that, when executed by a processor of a system, causes the system to:
extract, from an image, one or more visual attributes of the image using a first trained machine learning model;
map the one or more visual attributes to one or more emotion attributes using a second trained machine learning model;
generate a prompt to a large language model (LLM), the prompt being based on the one or more emotion attributes;
provide the generated prompt to the LLM; and
obtain, from the LLM, a generated description of the image.
