# US 2025/0094722 A1 · Annotation method for large language model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094722 A1
- Document ID: US20250094722A1
- Status: Published application
- Kind: A1
- Application number: 18/968,920
- Filed: 2024-12-04
- Published: 2025-03-20
- Earliest claimed: 2024-02-05
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Dai DAI; Hua Wu; Gangqiang Hu
- Web page: https://patentgrep.com/patent/US20250094722A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An annotation method for a large language model, an electronic device, and a medium are provided. The method may include: obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement; obtaining a plurality of scores corresponding to the plurality of response texts, where each of the plurality of scores indicates a degree to which a corresponding response text in the plurality of response texts matches the request text; and obtaining an annotated text for at least one of the plurality of response texts based on the plurality of scores, where the annotated text is used to adjust a parameter of the large language model.

## Claims

### Claim 1

An annotation method for a large language model, comprising:
obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement;
obtaining a plurality of scores corresponding to the plurality of response texts, wherein each score of the plurality of scores indicates a degree to which a response text corresponding to the score in the plurality of response texts matches the request text; and
obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores, wherein the annotated text is used to adjust a parameter of the large language model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises:
determining a first response text as the annotated text in response to determining that a score, corresponding to the first response text, in the plurality of scores satisfies a threshold score condition.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the obtaining a plurality of scores corresponding to the plurality of response texts comprises: obtaining, for each response text of the plurality of response texts, a level selected from a plurality of predetermined ordered levels as a score for the response text,
and wherein the score satisfying the threshold score condition corresponds to the highest level of the plurality of ordered levels.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises:
determining a second response text satisfying a modification condition from the plurality of response texts, in response to determining that none of the plurality of scores satisfy a threshold score condition; and
obtaining a modified version of the second response text as the annotated text.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises:
obtaining an evaluation text for the at least one response text of the plurality of response texts as the annotated text, in response to determining that none of the plurality of scores satisfy a threshold score condition and to determining that the plurality of response texts comprise no response text satisfying a modification condition.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the difference requirement indicates at least one of the following: a word segmentation difference and a reward model-based difference between the response texts.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising: before the obtaining a plurality of scores corresponding to the plurality of response texts,
obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking a degree to which each response text of the plurality of response texts matches the request text.

### Claim 9 (depends on 7)

The method according to claim 7, wherein the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking correctness of a fact recorded in each response text of the plurality of response texts.

### Claim 10 (depends on 7)

The method according to claim 7, wherein the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking language expression of each response text of the plurality of response texts.

### Claim 11 (depends on 7)

The method according to claim 7, wherein the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking correctness of logic of each response text of the plurality of response texts.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement comprises:
obtaining a response text set, wherein each response text in the response text set is generated by the large language model for the request text; and
selecting the plurality of response texts from the response text set based on the difference requirement.

### Claim 13

An electronic device, comprising:
one or more processors;
a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement;
obtaining a plurality of scores corresponding to the plurality of response texts, wherein each score of the plurality of scores indicates a degree to which a response text corresponding to the score in the plurality of response texts matches the request text; and
obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores, wherein the annotated text is used to adjust a parameter of the large language model.

### Claim 14 (depends on 13)

The electronic device according to claim 13, wherein the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises:
determining a first response text as the annotated text in response to determining that a score, corresponding to the first response text, in the plurality of scores satisfies a threshold score condition.

### Claim 15 (depends on 14)

The electronic device according to claim 14, wherein the obtaining a plurality of scores corresponding to the plurality of response texts comprises: obtaining, for each response text of the plurality of response texts, a level selected from a plurality of predetermined ordered levels as a score for the response text,
and wherein the score satisfying the threshold score condition corresponds to the highest level of the plurality of ordered levels.

### Claim 16 (depends on 14)

The electronic device according to claim 14, wherein the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises:
determining a second response text satisfying a modification condition from the plurality of response texts, in response to determining that none of the plurality of scores satisfy a threshold score condition; and
obtaining a modified version of the second response text as the annotated text.

### Claim 17 (depends on 16)

The electronic device according to claim 16, wherein the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises:
obtaining an evaluation text for the at least one response text of the plurality of response texts as the annotated text, in response to determining that none of the plurality of scores satisfy a threshold score condition and to determining that the plurality of response texts comprise no response text satisfying a modification condition.

### Claim 18 (depends on 13)

The electronic device according to claim 13, wherein the difference requirement indicates at least one of the following: a word segmentation difference and a reward model-based difference between the response texts.

### Claim 19 (depends on 13)

The electronic device according to claim 13, further comprising: before the obtaining a plurality of scores corresponding to the plurality of response texts, obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts.

### Claim 20

A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform processing comprising: obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement;
obtaining a plurality of scores corresponding to the plurality of response texts, wherein each of the plurality of scores indicates a degree to which a corresponding response text in the plurality of response texts matches the request text; and
obtaining an annotated text for at least one of the plurality of response texts based on the plurality of scores, wherein the annotated text is used to adjust a parameter of the large language model.
