Information retrieval system evaluation method, device and storage medium

An information retrieval system evaluation device, comprising:

• one or more processors, memory and one or more program units stored in the memory and to be executed by the one or more processors, the one or more program units comprising:
• a behavior data collecting unit, an analyzing unit, an evaluation retrieval unit and an evaluation indicator computing unit;
• wherein
• the behavior data collecting unit is configured to obtain behavior data sample reported by an information retrieval system within a pre-determined period;
• the analyzing unit is configured to obtain a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample;
• the evaluation retrieval unit is configured to invoke the information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result;
• and
• the evaluation indicator computing unit is configured to compute an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, wherein
• the recall rate is a ratio between the sum of all related object parameters in the evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in the sample retrieval result corresponding to the keyword;
• the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword;
• and
• the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword;
• the non-related object corresponding to the keyword is an object where a user does not perform operation in the sample retrieval result corresponding to the keyword.

A non-transitory storage medium containing a computer executable instruction, wherein the computer executable instruction is used to perform an information retrieval system evaluation method when executed by a compute processor, and the method comprises the following steps:

• obtaining behavior data sample reported by an information retrieval system within a pre-determined period;
• obtaining a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample;
• invoking an information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result;
• and
• computing an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, wherein
• the recall rate is a ratio between the sum of all the related object parameters in an evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in a sample retrieval result corresponding to the keyword;
• the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword;
• and
• the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword;
• the non-related object corresponding to the keyword is an object where a user not performs operation in the sample retrieval result corresponding to the keyword.