Multimodal data generation

A multimodal model training method, comprising:

• obtaining a sample data sequence, wherein the sample data sequence comprises at least one data segment, and wherein each data segment of the at least one data segment corresponds to one data modality;
• inputting the sample data sequence into a multimodal model, to obtain a plurality of tokens in a predicted data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following steps:
• in response to determining that the current token belongs to a first data modality, inputting the sample data sequence and a current predicted data sequence into the multimodal model, so that the multimodal model generates the current token based on the sample data sequence and the current predicted data sequence, wherein values of unit data of the first data modality are discrete;
• or
• in response to determining that the current token belongs to a second data modality, inputting the sample data sequence and the current predicted data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the sample data sequence and the current predicted data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token;
• and
• adjusting a parameter of the multimodal model based on a difference between the predicted data sequence and a target data sequence corresponding to the sample data sequence.

An electronic device, comprising:

• one or more processors;
• a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
• obtaining a query data sequence, wherein the query data sequence comprises at least one data segment, and each of the at least one data segment corresponds to one data modality;
• and
• inputting the query data sequence into a multimodal model, to obtain a plurality of tokens in a response data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following operations:
• in response to determining that the current token belongs to a first data modality, inputting the query data sequence and a current response data sequence into the multimodal model, so that the multimodal model generates the current token based on the query data sequence and the current response data sequence, wherein values of unit data of the first data modality are discrete;
• or
• in response to determining that the current token belongs to a second data modality, inputting the query data sequence and the current response data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the query data sequence and the current response data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token.