Automatic evaluation of a knowledge canvassing application

A system, comprising:

• a machine-learning knowledge canvassing system;
• a processor;
• and
• a memory coupled to the processor, the memory encoded with instructions that when executed cause the processor to:
• generate benchmark data, each benchmark datum including a set of one or more benchmark input entities and a set of one or more benchmark output entities associated with the one or more benchmark input entities;
• query the machine-learning knowledge canvassing system with each set of benchmark input entities;
• receive a returned passage from the machine-learning knowledge canvassing system in response to the query;
• receive, for each set of benchmark input entities queried, an output result from the machine-learning knowledge canvassing system that includes a set of zero or more knowledge canvassing system output entities;
• generate an evaluation score for each set of knowledge canvassing system output entities based on a comparison of the knowledge canvassing system output entities with the set of benchmark output entities, wherein the comparison includes an estimation of a degree to which a knowledge canvassing system output entity in a first set of the knowledge canvassing system output entities relates to a benchmark output entity in a first set of the benchmark output entities, wherein the estimation of the degree to which the knowledge canvassing system output entity relates to the benchmark output entity is based on a co-occurrence, in a returned passage from the machine-learning knowledge canvassing system, of the knowledge canvassing system output entity and the benchmark output entity, and wherein the knowledge canvassing system output entity does not match any of the benchmark output entities in the first set of benchmark output entities;
• and
• train the machine-learning knowledge canvassing system based on the evaluation scores.

A computer program product for training a machine-learning knowledge canvassing system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

• generate benchmark data, each benchmark datum including a set of one or more benchmark input entities and a set of one or more benchmark output entities associated with the one or more benchmark input entities;
• query the machine-learning knowledge canvassing system with each set of benchmark input entities;
• receive a returned passage from the machine-learning knowledge canvassing system in response to the query;
• receive, for each set of benchmark input entities queried, an output result from the machine-learning knowledge canvassing system that includes a set of zero or more knowledge canvassing system output entities;
• generate an evaluation score for each set of knowledge canvassing system output entities based on a comparison of the knowledge canvassing system output entities with the set of benchmark output entities, wherein the comparison includes an estimation of a degree to which a knowledge canvassing system output entity in a first set of the knowledge canvassing system output entities relates to a benchmark output entity in a first set of the benchmark output entities, wherein the estimation of the degree to which the knowledge canvassing system output entity relates to the benchmark output entity is based on a co-occurrence, in a returned passage from the machine-learning knowledge canvassing system, of the knowledge canvassing system output entity and the benchmark output entity, and wherein the knowledge canvassing system output entity does not match any of the benchmark output entities in the first set of benchmark output entities;
• and
• train the machine-learning knowledge canvassing system based on the evaluation scores.