Continual learning method and device for multimodality data

A computing device comprising:

• a processor;
• and
• a memory storing one or more programs executed by the processor,
• wherein the processor performs a continual learning method, and
• the processor is configured to perform operations comprising:
• receiving multimodality data including a plurality of data items having different modalities and tokenizing each piece of data;
• receiving text data representing a class for the multimodality data and tokenizing the text data;
• generating an aggregated modality prompt for the multimodality data;
• generating an aggregated text prompt for the text data;
• inputting modality concatenation data, in which the aggregated modality prompt is concatenated with the tokenized multimodality data, into a vision encoder and outputting a modality embedding vector;
• inputting text concatenation data in which the aggregated text prompt is concatenated with the tokenized text data into a language encoder and outputting a text embedding vector;
• and
• projecting the text embedding vector into an embedding space through a projection head of the language encoder and projecting the modality embedding vector into the embedding space through a projection head of the vision encoder so that the modality embedding vector and the text embedding vector that correspond to each other are matched.

A computer program stored in a non-transitory computer readable storage medium, comprising:

• one or more instructions,
• wherein the instructions, when executed by a computing device including one or more processors, cause the computing device to perform operations comprising:
• receiving multimodality data including a plurality of data items having different modalities and tokenizing each piece of data;
• receiving text data representing a class for the multimodality data and tokenizing the text data;
• generating an aggregated modality prompt for the multimodality data;
• generating an aggregated text prompt for the text data;
• inputting modality concatenation data, in which the aggregated modality prompt is concatenated with the tokenized multimodality data, into a vision encoder and outputting a modality embedding vector;
• inputting text concatenation data in which the aggregated text prompt is concatenated with the tokenized text data into a language encoder and outputting a text embedding vector;
• and
• projecting the text embedding vector into an embedding space through a projection head of the language encoder and projecting the modality embedding vector into the embedding space through a projection head of the vision encoder so that the modality embedding vector and the text embedding vector that correspond to each other are matched.