# US 2024/0177507 A1 · Apparatus and method for generating text from image and method of training model for generating text from image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177507 A1
- Document ID: US20240177507A1
- Status: Published application
- Kind: A1
- Application number: 18/499,717
- Filed: 2023-11-01
- Published: 2024-05-30
- Earliest claimed: 2022-11-24
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Dong Hyuck IM; Jung Hyun KIM; Hye Mi KIM; Jee Hyun PARK; Yong Seok SEO; Won Young YOO
- Web page: https://patentgrep.com/patent/US20240177507A1

## CPC

- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An apparatus for generating text from an image may comprise: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction, wherein the processor is further configured to generate encoding information for an image based on the image and extract text information related to content of the image based on a degree of association with the encoding information.

## Claims

### Claim 1

An apparatus for generating text from an image, the apparatus comprising:
a memory configured to store at least one instruction; and
a processor configured to execute the at least one instruction,
wherein the processor, by the at least one instruction, is further configured to:
generate encoding information for an image based on the image; and
extract text information related to content of the image based on a degree of association with the encoding information.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to extract the text information by translating the encoding information into text.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to extract, as the text information, text having high attention with the encoding information as the degree of association with the encoding information.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to generate the encoding information using a codebook for encoding the image.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to generate the encoding information by converting the image into an image token sequence.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to generate image-text synthetic data for the image by combining the image and the text information.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the processor is further configured to generate an image based on the text using the image-text synthetic data.

### Claim 8

A method of generating text from an image, which is performed by a processor configured to execute at least one instruction stored in a memory when the processor executes the at least one instruction, the method comprising:
generating encoding information for an image based on the image; and
extracting text information related to content of the image based on a degree of association with the encoding information.

### Claim 9 (depends on 8)

The method of claim 8, wherein, in the extracting of the text information, the encoding information is translated into text so that the text information is extracted.

### Claim 10 (depends on 8)

The method of claim 8, wherein, in the extracting of the text information, text having high attention with the encoding information as the degree of association with the encoding information is extracted as the text information.

### Claim 11 (depends on 8)

The method of claim 8, wherein, in the generating of the encoding information, the encoding information is generated using a codebook for encoding the image.

### Claim 12 (depends on 8)

The method of claim 8, wherein, in the generating of the encoding information, the image is converted into an image token sequence so that the encoding information is generated.

### Claim 13 (depends on 8)

The method of claim 8, further comprising combining the image and the text information to generate image-text synthetic data for the image.

### Claim 14 (depends on 13)

The method of claim 13, further comprising generating an image based on the text using the image-text synthetic data.

### Claim 15

A method of training a model for generating text from an image, which is performed by a processor configured to execute at least one instruction stored in a memory when the processor executes the at least one instruction, the method comprising:
generating encoding information for an image based on the image; and
controlling an image-text translation model to learn a function of extracting text information related to content of the image based on a degree of association with the encoding information.

### Claim 16 (depends on 15)

The method of claim 15, wherein, in the learning by the image-text translation model, the image-text translation model learns a function of extracting, as the text information, text having high attention with the encoding information as the degree of association with the encoding information.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
inputting a training image to an encoder-decoder model; and
controlling the encoder-decoder model so that the encoder-decoder model learns a function of generating an image codebook from the training image.

### Claim 18 (depends on 17)

The method of claim 17, wherein, in the generating of the encoding information, the image is converted into an image token sequence using the image codebook so that the encoding information is generated.

### Claim 19 (depends on 15)

The method of claim 15, further comprising combining the image and the text information to generate image-text synthetic data for the image.

### Claim 20 (depends on 19)

The method of claim 19, further comprising controlling a text-based image generation model so that the text-based image generation model learns a function of generating an image based on the text using the image-text synthetic data.
