US 2024/0176959 A1 · METHOD AND APPARATUS FOR GENERATING LANGUAGE MODEL USING CROSSMODAL INFORMATION

Abstract

Provided is a method of generating a language model using crossmodal information. The method includes: receiving language-based first modality information and non-language-based second modality information; converting the first modality information into a first byte sequence; converting the second modality information into a second byte sequence; converting the first and second byte sequences into a first embedding vector and a second embedding vector by applying an embedding technique for each modality; generating semantic association information between first and second modality information by inputting the first and second embedding vectors to a crossmodal transformer; and learning the language model by setting the generated semantic association information as training data.

Claims14

Dependent claims

A method of generating a language model using crossmodal information, comprising:

receiving language-based first modality information and non-language-based second modality information;
converting the first modality information into a first byte sequence;
converting the second modality information into a second byte sequence;
converting the first and second byte sequences into a first embedding vector and a second embedding vector by applying an embedding technique for each modality;
generating semantic association information between the first and second modality information by inputting the first and second embedding vectors to a crossmodal transformer;
and
learning the language model by setting the generated semantic association information as training data.

4 more elements

An apparatus for generating a language model using crossmodal information, comprising:

a communication unit configured to receive language-based first modality information and non-language-based second modality information;
a memory configured to store a program for learning the language model based on semantic association information between the first and second modality information;
and
as the program stored in the memory is executed, a processor configured to convert the first and second modality information into first and second byte sequences, respectively, convert the first and second byte sequences into a first embedding vector and a second embedding vector by applying an embedding technique for each modality, generate semantic association information between first and second modality information by inputting the first and second embedding vectors to a crossmodal transformer, and learn the language model by setting the generated semantic association information as training data.

1 more element

Dependencies are inferred from the claim text.

Method and apparatus for generating language model using crossmodal information

Life of this patent

Method and apparatus for generating language model using crossmodal information

Life of this patent

Abstract

Claims14