# US 9,436,729 B2 · Information retrieval system evaluation method, device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,436,729 B2
- Document ID: US9436729B2
- Status: Granted
- Kind: B2
- Application number: 14/328,809
- Filed: 2014-07-11
- Published: 2014-10-30
- Granted: 2016-09-06
- Earliest claimed: 2013-03-15
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Yan Shen; Ruxuan Xue; Zhengping Zhu
- Web page: https://patentgrep.com/patent/US9436729B2

## CPC

- G06F 16/245: Query processing
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The present disclosure discloses an information retrieval system evaluation method, device and storage medium. A ratio between the sum of all the related object parameters of a keyword in keyword in an evaluation retrieval result set and the sum of all the related object parameters of the keyword in keyword in a retrieval result set is used to compute a recall rate of an information retrieval system. And the recall rate is introduced to evaluate the information retrieval system, thereby enhancing accuracy of quantitative evaluation of the information retrieval system, and improving the automation degree of evaluation.

## Claims

### Claim 1

An information retrieval system evaluation method, comprising:
Obtaining, by an evaluation electronic device, behavior data sample reported by an information retrieval system within a pre-determined period;
obtaining, by an evaluation electronic device, a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample;
invoking, by an evaluation electronic device, the information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and
computing, by an evaluation electronic device, an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, where
the recall rate is a ratio between the sum of all related object parameters in the evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in the sample retrieval result corresponding to the keyword;
the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and
the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user does not perform operation in the sample retrieval result corresponding to the keyword.

### Claim 2 (depends on 1)

The information retrieval system evaluation method according to claim 1, wherein the evaluation indicator of the information retrieval system is computed according to the following formula, by an evaluation electronic device:
F
=
(
β
2
+
1
)
⁢
PR
β
2
⁢
P
+
R
wherein, F is the evaluation indicator, R is an arithmetic mean value of recall rates corresponding to all evaluation keywords, P is an arithmetic mean value of correctness percentages corresponding to all evaluation keywords, and β is a pre-determined weight coefficient.

### Claim 3 (depends on 2)

The information retrieval system evaluation method according to claim 2, wherein the arithmetic mean value of the recall rates is computed according to the following formula, by an evaluation electronic device:
R
=
1
n
⁢
∑
k
=
1
n
⁢
⁢
R
k
,
R
k
=
HitScore
k
TotalScore
k
wherein, n is the number of keywords in the evaluation keyword set, Rk is a recall rate of a Kth evaluation keyword, HitScorek is the sum of all the related object parameters in an evaluation retrieval result corresponding to a Kth evaluation keyword, and TotalScorek is the sum of all the related object parameters in a sample retrieval result corresponding to a Kth evaluation keyword, wherein a related object parameter corresponding to the Kth evaluation keyword is computed according to the number of times that a user performs operation on a related object corresponding to the Kth evaluation keyword and an arrangement position of a related object corresponding to the Kth evaluation keyword in its each corresponding sample retrieval sub-result.

### Claim 4 (depends on 3)

The information retrieval system evaluation method to claim 3, wherein the related object parameter score of each related object is computed according to the following formula, by an evaluation electronic device:
score
=
a
ExposeCnt
⁢
∑
x
=
1
FCnt
⁢
⁢
FirstScore
x
+
b
ExposeCnt
⁢
∑
y
=
1
SCnt
⁢
⁢
SecondScore
y
+
σ
wherein, ExposeCnt is the total number of times that the related object appears in a sample retrieval result of a corresponding keyword, FCnt is the total number of times that a user performs first operation on the related object appearing in a sample retrieval result of a corresponding keyword, SCnt is the total number of times that a user performs second operation on the related object appearing in a sample retrieval result of a corresponding keyword, a is a first operation weight coefficient, b is a second operation weight coefficient, σ is an offset coefficient;
FirstScorex is a dependency coefficient when the related object appears in the sample retrieval result of the corresponding keyword for an xth time and being performed with the first operation, the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval; and SecondScorey is a dependency coefficient when the related object appears in the sample retrieval result of the corresponding keyword for a yth time and being performed with the second operation, wherein the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval.

### Claim 5 (depends on 2)

The information retrieval system evaluation method according to claim 2, wherein the arithmetic mean value of the correctness percentages is computed according to the following formula, by an evaluation electronic device:
P
=
1
n
⁢
∑
k
=
1
n
⁢
⁢
P
k
,
P
k
=
1
H
k
⁢
∑
r
=
1
H
k
⁢
⁢
H
k
-
I
k
,
r
H
k
wherein, n is the number of keywords in the evaluation keyword set, Pk is the correctness percentage of a Kth evaluation keyword, Hk is the number of the related objects included in the evaluation retrieval result of the Kth evaluation keyword, Ik,r is the number of the non-related objects whose arrangement position is in front of the arrangement position of an rth related object in the evaluation retrieval result of the Kth evaluation keyword.

### Claim 6

An information retrieval system evaluation device, comprising: one or more processors, memory and one or more program units stored in the memory and to be executed by the one or more processors, the one or more program units comprising: a behavior data collecting unit, an analyzing unit, an evaluation retrieval unit and an evaluation indicator computing unit; wherein
the behavior data collecting unit is configured to obtain behavior data sample reported by an information retrieval system within a pre-determined period;
the analyzing unit is configured to obtain a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample;
the evaluation retrieval unit is configured to invoke the information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and
the evaluation indicator computing unit is configured to compute an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, wherein
the recall rate is a ratio between the sum of all related object parameters in the evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in the sample retrieval result corresponding to the keyword;
the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and
the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user does not perform operation in the sample retrieval result corresponding to the keyword.

### Claim 7 (depends on 6)

The information retrieval system evaluation device according to claim 6, wherein, the evaluation indicator computing unit is configured to compute the evaluation indicator of the information retrieval system according to the following formula:
F
=
(
β
2
+
1
)
⁢
PR
β
2
⁢
P
+
R
wherein, F is the evaluation indicator, R is an arithmetic mean value of recall rates corresponding to all evaluation keywords, P is an arithmetic mean value of correctness percentages corresponding to all evaluation keywords, and β is a pre-determined weight coefficient.

### Claim 8 (depends on 7)

The information retrieval system evaluation device according to claim 7, wherein the evaluation retrieval unit comprises a recall rate computing subunit; wherein
the recall rate computing subunit is configured to compute an arithmetic mean value of the recall rates according to the following formula:
R
=
1
n
⁢
∑
k
=
1
n
⁢
⁢
R
k
,
R
k
=
HitScore
k
TotalScore
k
wherein, n is the number of keywords in the evaluation keyword set, Rk is a recall rate of a Kth evaluation keyword, HitScorek is the sum of all the related object parameters in an evaluation retrieval result corresponding to a Kth evaluation keyword, and TotalScorek is the sum of all the related object parameters in a sample retrieval result corresponding to a Kth evaluation keyword, wherein a related object parameter corresponding to the Kth evaluation keyword is computed according to the number of times that a user performs operation on a related object corresponding to the Kth evaluation keyword and an arrangement position of a related object corresponding to the Kth evaluation keyword in its each corresponding sample retrieval sub-result.

### Claim 9 (depends on 8)

The information retrieval system evaluation device according to claim 8, wherein the analyzing unit comprises a related object parameter computing subunit; wherein
the related object parameter subunit is configured to compute a corresponding related object parameter score according to the following formula:
score
=
a
ExposeCnt
⁢
∑
x
=
1
FCnt
⁢
⁢
FirstScore
x
+
b
ExposeCnt
⁢
∑
y
=
1
SCnt
⁢
⁢
SecondScore
y
+
σ
wherein, ExposeCnt is the total number of times that the related object appears in a sample retrieval result of a corresponding keyword, FCnt is the total number of times that a user performs first operation on the related object appears in a sample retrieval result of a corresponding keyword, SCnt is the total number of times that a user performs second operation on the related object appearing in a sample retrieval result of a corresponding keyword, a is a first operation weight coefficient, b is a second operation weight coefficient, σ is an offset coefficient; FirstScorex is a dependency coefficient when the related object appearing in the sample retrieval result of the corresponding keyword for an xth time and being performed with the first operation, the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval; and SecondScorey is a dependency coefficient when the related object appears in the sample retrieval result of the corresponding keyword for a yth time and being performed with the second operation, wherein the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval.

### Claim 10 (depends on 7)

The information retrieval system evaluation device according to claim 7, wherein the evaluation retrieval unit comprises a correctness percentage computing subunit; wherein
the correctness percentage computing subunit is configured to compute the arithmetic mean value of the correctness percentages according to the following formula:
P
=
1
n
⁢
∑
k
=
1
n
⁢
⁢
P
k
,
P
k
=
1
H
k
⁢
∑
r
=
1
H
k
⁢
⁢
H
k
-
I
k
,
r
H
k
wherein, n is the number of keywords in the evaluation keyword set, Pk is the correctness percentage of a Kth evaluation keyword, Hk is the number of the related objects included in the evaluation retrieval result of the Kth evaluation keyword, and Ik,r is the number of the non-related objects whose arrangement position is in front of the arrangement position of an rth related object in the evaluation retrieval result of the Kth evaluation keyword.

### Claim 11

A non-transitory storage medium containing a computer executable instruction, wherein the computer executable instruction is used to perform an information retrieval system evaluation method when executed by a compute processor, and the method comprises the following steps:
obtaining behavior data sample reported by an information retrieval system within a pre-determined period;
obtaining a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample;
invoking an information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and
computing an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, wherein
the recall rate is a ratio between the sum of all the related object parameters in an evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in a sample retrieval result corresponding to the keyword;
the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and
the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user not performs operation in the sample retrieval result corresponding to the keyword.