Apparatus and method for generating text from image and method of training model for generating text from image
AssigneeElectronics and Telecommunications Research Institute
InventorsDong Hyuck Im, Jung Hyun Kim, Hye Mi Kim, Jee Hyun Park, Yong Seok Seo, Won Young Yoo
Life of this patent
Estimating the term…
Abstract
An apparatus for generating text from an image may comprise: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction, wherein the processor is further configured to generate encoding information for an image based on the image and extract text information related to content of the image based on a degree of association with the encoding information.
Claims20
An apparatus for generating text from an image, the apparatus comprising:
- a memory configured to store at least one instruction;
- and
- a processor configured to execute the at least one instruction,
- wherein the processor is further configured to:
- convert the image into an image token sequence using a learned codebook;
- generate encoding information comprising the image token sequence;
- and
- generate text information describing semantic content of the image by applying an image-to-text translation model that receives the encoding information as an input sequence, and
- wherein the image-to-text translation model is configured to generate the text information based on learned degrees of association between image tokens in the encoding information and text tokens.
A method of generating text from an image performed by a processor executing at least one instruction stored in a memory, the method comprising:
- converting the image into an image token sequence using a learned codebook;
- generating encoding information comprising the image token sequence;
- and
- generating text information describing semantic content of the image by applying an image-to-text translation model that receives the encoding information as an input sequence,
- wherein the image-to-text translation model is configured to generate the text information based on learned degrees of association between image tokens in the encoding information and text tokens.
A method of training an image-to-text translation model for generating text describing semantic content of an image, performed by a processor executing at least one instruction stored in a memory, the method comprising:
- converting an image into an image token sequence using a learned codebook;
- generating encoding information comprising the image token sequence;
- and
- training the image-to-text translation model to learn a function for generating text information describing semantic content of the image using the encoding information as an input sequence,
- wherein the function enables the image-to-text translation model to generate the text information based on learned degrees of association between image tokens in the encoding information and text tokens.
Dependencies are inferred from the claim text.