# US 2025/0094713 A1 · Multimodal data generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094713 A1
- Document ID: US20250094713A1
- Status: Published application
- Kind: A1
- Application number: 18/967,529
- Filed: 2024-12-03
- Published: 2025-03-20
- Earliest claimed: 2024-04-01
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Shuohuan WANG; Yekun CHAI; Siyu DING; Junyuan SHANG; Zhenyu ZHANG; Yu SUN; Hao TIAN; Hua WU; Haifeng WANG
- Web page: https://patentgrep.com/patent/US20250094713A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 16/3329: Natural language query formulation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A multimodal data generation method is provided. The method includes: inputting a query data sequence into a multimodal model, to obtain a plurality of tokens in a response data sequence, where a current token is generated through the following operations: inputting the query data sequence and a current response data sequence into the multimodal model, so that the multimodal model generates the current token based on the query data sequence and the current response data sequence, in response to determining that the current token belongs to a first data modality; or inputting the query data sequence and a current response data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the query data sequence and the current response data sequence, to generate a result token sequence, in response to determining that the current token belongs to a second data modality.

## Claims

### Claim 1

A multimodal data generation method, comprising:
obtaining a query data sequence, wherein the query data sequence comprises at least one data segment, and wherein each data segment of the at least one data segment corresponds to one data modality; and
inputting the query data sequence into a multimodal model, to obtain a plurality of tokens in a response data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following operations:
in response to determining that the current token belongs to a first data modality, inputting the query data sequence and a current response data sequence into the multimodal model, so that the multimodal model generates the current token based on the query data sequence and the current response data sequence, wherein values of unit data of the first data modality are discrete; or
in response to determining that the current token belongs to a second data modality, inputting the query data sequence and the current response data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the query data sequence and the current response data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the plurality of tokens comprise a modality tag pair, the modality tag pair comprising a modality data start tag and a modality data end tag that indicate a same data modality, and there being at least one token belonging to a corresponding data modality between the modality data start tag and the modality data end tag, and the method further comprises:
determining a data modality to which the current token belongs based on a last modality data start tag in the current response data sequence.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the determining a data modality to which the current token belongs based on a last modality data start tag in the current response data sequence comprises:
in response to determining that the last modality data start tag in the current response data sequence indicates the first data modality, determining that the current token belongs to the first data modality; or
in response to determining that the last modality data start tag in the current response data sequence indicates the second data modality, determining that the current token belongs to the second data modality.

### Claim 4 (depends on 3)

The method according to claim 3, further comprising:
after that the result token sequence is generated, sequentially generating a modality data end tag indicating the second data modality, and a modality data start tag indicating the first data modality.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the modality data start tag and the modality data end tag are both tokens of the first data modality, and the method further comprises:
in response to determining that the current response data sequence is empty, determining that the current token belongs to the first data modality.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the denoising, through the multimodal model, an initial token sequence based on the query data sequence and the current response data sequence, to generate a result token sequence comprises:
performing, through the multimodal model, a preset number of reverse diffusion operations on the initial token sequence based on the query data sequence and the current response data sequence, to generate the result token sequence, wherein each reverse diffusion operation comprises:
inputting the query data sequence, the current response data sequence, a current token sequence, and an identifier of a current reverse diffusion round into the multimodal model, so that the multimodal model generates a denoised token sequence,
wherein a current token sequence for a first reverse diffusion operation is the initial token sequence, a current token sequence for a second and each subsequent reverse diffusion operation is a denoised token sequence generated by a previous reverse diffusion operation, and a denoised token sequence generated by a last reverse diffusion operation is the result token sequence.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the preset quantity is a token of the first data modality that is output by the multimodal model.

### Claim 8 (depends on 6)

The method according to claim 6, wherein the preset number is a token of the first data modality that is output by the multimodal model.

### Claim 9 (depends on 6)

The method according to claim 6, wherein the preset quantity and the preset number are both tokens of the first data modality that are output by the multimodal model.

### Claim 10 (depends on 1)

The method according to claim 1, further comprising:
obtaining an initial query data sequence entered by a user, wherein the initial query data sequence comprises at least one initial data segment, and wherein each initial data segment of the at least one initial data segment corresponds to one data modality; and
adding, for each initial data segment of the at least one initial data segment, a modality data start tag indicating a data modality of the initial data segment before the initial data segment, and adding a modality data end tag indicating a data modality of the initial data segment after the initial data segment, to obtain the query data sequence.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the multimodal model is a transformer model that comprises only a decoder.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the first data modality comprises any one of the following: natural language text, tables, code, SMILES molecular formulas, or protein sequences; and
the second data modality comprises any one of the following: images, videos, audio, or point cloud data.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the initial token sequence is a random noise sequence.

### Claim 14

A multimodal model training method, comprising:
obtaining a sample data sequence, wherein the sample data sequence comprises at least one data segment, and wherein each data segment of the at least one data segment corresponds to one data modality;
inputting the sample data sequence into a multimodal model, to obtain a plurality of tokens in a predicted data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following steps:
in response to determining that the current token belongs to a first data modality, inputting the sample data sequence and a current predicted data sequence into the multimodal model, so that the multimodal model generates the current token based on the sample data sequence and the current predicted data sequence, wherein values of unit data of the first data modality are discrete; or
in response to determining that the current token belongs to a second data modality, inputting the sample data sequence and the current predicted data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the sample data sequence and the current predicted data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token;
and
adjusting a parameter of the multimodal model based on a difference between the predicted data sequence and a target data sequence corresponding to the sample data sequence.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the sample data sequence comprises at least two of the following:
a first sample data sequence that only comprises a data segment of the first data modality;
a second sample data sequence that only comprises a data segment of the second data modality, or
a third sample data sequence that comprises interleaved data segments of the first data modality and the second data modality.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the method is performed by a plurality of electronic devices, any one of the plurality of electronic devices is configured to train the multimodal model by using sample data sequences of a same type, and a loss value of the multimodal model is a sum of respective local loss values of the plurality of electronic devices.

### Claim 17 (depends on 14)

The method according to claim 14, wherein the sample data sequence only comprises a data segment of the first data modality, and wherein, the target data sequence is the sample data sequence itself.

### Claim 18 (depends on 14)

The method according to claim 14, wherein the sample data sequence comprises at least a data segment of the second data modality, and the method further comprises:
obtaining a first data sequence, wherein the first data sequence comprises at least a first data segment of the second data modality;
performing a plurality of noise addition operations on the first data segment, to obtain a plurality of second data segments corresponding to the plurality of noise addition operations respectively, thereby obtaining a plurality of second data sequences corresponding to the plurality of noise addition operations respectively,
wherein the sample data sequence is any one of the plurality of second data sequences, and the target data sequence is the first data sequence.

### Claim 19 (depends on 18)

The method according to claim 18, wherein the initial token sequence is obtained by segmenting a second data segment in the sample data sequence.

### Claim 20

An electronic device, comprising:
one or more processors;
a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining a query data sequence, wherein the query data sequence comprises at least one data segment, and each of the at least one data segment corresponds to one data modality; and
inputting the query data sequence into a multimodal model, to obtain a plurality of tokens in a response data sequence output sequentially by the multimodal model, wherein a current token among the plurality of tokens is generated through the following operations:
in response to determining that the current token belongs to a first data modality, inputting the query data sequence and a current response data sequence into the multimodal model, so that the multimodal model generates the current token based on the query data sequence and the current response data sequence, wherein values of unit data of the first data modality are discrete; or
in response to determining that the current token belongs to a second data modality, inputting the query data sequence and the current response data sequence into the multimodal model, so that the multimodal model denoises an initial token sequence based on the query data sequence and the current response data sequence, to generate a result token sequence, wherein values of unit data of the second data modality are continuous, wherein the initial token sequence comprises a preset quantity of initial tokens, and wherein the result token sequence comprises the preset quantity of tokens starting from the current token.
