# US 2025/0094789 A1 · Method for evaluating large model, electronic device and computer readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094789 A1
- Document ID: US20250094789A1
- Status: Published application
- Kind: A1
- Application number: 18/968,810
- Filed: 2024-12-04
- Published: 2025-03-20
- Earliest claimed: 2024-09-18
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Hua LU; Shilong FAN; Zeyang LEI; Bingjin CHEN; Siqi BAO; Hua WU
- Web page: https://patentgrep.com/patent/US20250094789A1

## CPC

- G06F 40/30: Semantic analysis
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for evaluating a large model, an electronic device and a computer readable storage medium are provided, which relate to a field of artificial intelligence technology, and in particular to fields of large models technology and deep learning technology. The method includes: evaluating a response information of each of M large language models for an input instruction based on a preset evaluation rule, so as to obtain a first evaluation information for each response information, where M is a positive integer greater than 1; evaluating, in response to the first evaluation information for the M large language models being consistent with each other, each response information in a plurality of evaluation dimensions, so as to obtain a second evaluation information for each response information; and determining an evaluation result representing a responsiveness of each large language model, according to the second evaluation information for each response information.

## Claims

### Claim 1

A method for evaluating a large model, comprising:
evaluating a response information of each of M large language models for an input instruction based on a preset evaluation rule, so as to obtain a first evaluation information for each response information, where M is a positive integer greater than 1;
evaluating, in response to the first evaluation information for the M large language models being consistent with each other, each response information in a plurality of evaluation dimensions, so as to obtain a second evaluation information for each response information; and
determining an evaluation result representing a responsiveness of each of the M large language models, according to the second evaluation information for each response information.

### Claim 2 (depends on 1)

The method of claim 1, wherein each large language model is provided with a prompt information for guiding the large language model to respond to the input instruction; and
wherein the preset evaluation rule comprises at least one of:
a priority of the prompt information is higher than a priority of the input instruction; or
the priority of the input instruction is higher than a priority of the response information.

### Claim 3 (depends on 2)

The method of claim 2, wherein the evaluating a response information of each of M large language models for an input instruction based on a preset evaluation rule to obtain a first evaluation information for each response information comprises: for each large language model,
determining a first evaluation information indicating that the large language model meets the preset evaluation rule, if the response information is consistent with the prompt information; and
determining a first evaluation information indicating that the large language model does not meet the preset evaluation rule, if the response information is inconsistent with the prompt information.

### Claim 4 (depends on 1)

The method of claim 1, wherein the evaluating each response information in a plurality of evaluation dimensions to obtain a second evaluation information for each response information comprises: for each response information,
performing, based on a prompt information for each evaluation dimension, semantic matching on the response information to obtain a matching information for each evaluation dimension; and
determining the second evaluation information for the response information according to the matching information for each evaluation dimension.

### Claim 5 (depends on 4)

The method of claim 4, wherein the prompt information for each evaluation dimension comprises at least one of: a persona customization information, a role customization information, a capability customization information or a style customization information.

### Claim 6 (depends on 1)

The method of claim 1, wherein the determining an evaluation result representing a responsiveness of each of the M large language models, according to the second evaluation information for each response information comprises:
weighting the second evaluation information for each response information according to a preset weight for each evaluation dimension, so as to obtain a weighted evaluation information for each response information;
ranking the M large language models according to the weighted evaluation information for each response information, so as to obtain ranked M large language models; and
determining the large language models ranked in and before a predetermined ranking as having a first level of responsiveness, and determining other large language models as having a second level of responsiveness, so as to obtain the evaluation result.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
in response to the first evaluation information for the M large language models being inconsistent, determining the large language model with the first evaluation information indicating that the large language model meets the preset evaluation rule as having a first level of responsiveness; and
determining the large language model with the first evaluation information indicating that the large language model does not meet the preset evaluation rule as having a second level of responsiveness.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
evaluating the response information of the large language model for the input instruction based on the preset evaluation rule, to obtain the first evaluation information;
determining the large language model as having a first level of responsiveness, if the first evaluation information indicates that the large language model meets the preset evaluation rule; and
determining the large language model as having a second level of responsiveness, if the first evaluation information indicates that the large language model does not meet the preset evaluation rule.

### Claim 9

An electronic device, comprising:
one or more processors; and
a memory configured to store one or more computer programs,
wherein the one or more processors are configured to execute the one or more computer programs to:
evaluate a response information of each of M large language models for an input instruction based on a preset evaluation rule, so as to obtain a first evaluation information for each response information, where M is a positive integer greater than 1;
evaluate, in response to the first evaluation information for the M large language models being consistent with each other, each response information in a plurality of evaluation dimensions, so as to obtain a second evaluation information for each response information; and
determine an evaluation result representing a responsiveness of each of the M large language models, according to the second evaluation information for each response information.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein each large language model is provided with a prompt information for guiding the large language model to respond to the input instruction; and
wherein the preset evaluation rule comprises at least one of:
a priority of the prompt information is higher than a priority of the input instruction; or
the priority of the input instruction is higher than a priority of the response information.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the one or more processors are further configured to: for each large language model,
determine a first evaluation information indicating that the large language model meets the preset evaluation rule, if the response information is consistent with the prompt information; and
determine a first evaluation information indicating that the large language model does not meet the preset evaluation rule, if the response information is inconsistent with the prompt information.

### Claim 12 (depends on 9)

The electronic device of claim 9, wherein the one or more processors are further configured to: for each response information,
perform, based on a prompt information for each evaluation dimension, semantic matching on the response information to obtain a matching information for each evaluation dimension; and
determine the second evaluation information for the response information according to the matching information for each evaluation dimension.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the prompt information for each evaluation dimension comprises at least one of: a persona customization information, a role customization information, a capability customization information or a style customization information.

### Claim 14 (depends on 9)

The electronic device of claim 9, wherein the one or more processors are further configured to:
weight the second evaluation information for each response information according to a preset weight for each evaluation dimension, so as to obtain a weighted evaluation information for each response information;
rank the M large language models according to the weighted evaluation information for each response information, so as to obtain ranked M large language models; and
determine the large language models ranked in and before a predetermined ranking as having a first level of responsiveness, and determining other large language models as having a second level of responsiveness, so as to obtain the evaluation result.

### Claim 15 (depends on 9)

The electronic device of claim 9, wherein the one or more processors are further configured to:
in response to the first evaluation information for the M large language models being inconsistent, determine the large language model with the first evaluation information indicating that the large language model meets the preset evaluation rule as having a first level of responsiveness; and
determine the large language model with the first evaluation information indicating that the large language model does not meet the preset evaluation rule as having a second level of responsiveness.

### Claim 16 (depends on 9)

The electronic device of claim 9, wherein the one or more processors are further configured to:
evaluate the response information of the large language model for the input instruction based on the preset evaluation rule, to obtain the first evaluation information;
determine the large language model as having a first level of responsiveness, if the first evaluation information indicates that the large language model meets the preset evaluation rule; and
determine the large language model as having a second level of responsiveness, if the first evaluation information indicates that the large language model does not meet the preset evaluation rule.

### Claim 17

A computer readable storage medium storing computer programs or instructions, wherein the computer programs or instructions, when executed by a processor, cause the processor to:
evaluate a response information of each of M large language models for an input instruction based on a preset evaluation rule, so as to obtain a first evaluation information for each response information, where M is a positive integer greater than 1;
evaluate, in response to the first evaluation information for the M large language models being consistent with each other, each response information in a plurality of evaluation dimensions, so as to obtain a second evaluation information for each response information; and
determine an evaluation result representing a responsiveness of each of the M large language models, according to the second evaluation information for each response information.

### Claim 18 (depends on 17)

The computer readable storage medium of claim 17, wherein each large language model is provided with a prompt information for guiding the large language model to respond to the input instruction; and
wherein the preset evaluation rule comprises at least one of:
a priority of the prompt information is higher than a priority of the input instruction; or
the priority of the input instruction is higher than a priority of the response information.

### Claim 19 (depends on 18)

The computer readable storage medium of claim 18, wherein the computer programs or instructions are further configured to cause the processor to: for each large language model,
determine a first evaluation information indicating that the large language model meets the preset evaluation rule, if the response information is consistent with the prompt information; and
determine a first evaluation information indicating that the large language model does not meet the preset evaluation rule, if the response information is inconsistent with the prompt information.

### Claim 20 (depends on 17)

The computer readable storage medium of claim 17, wherein the computer programs or instructions are further configured to cause the processor to: for each response information,
perform, based on a prompt information for each evaluation dimension, semantic matching on the response information to obtain a matching information for each evaluation dimension; and
determine the second evaluation information for the response information according to the matching information for each evaluation dimension.
