Summary evaluation device, method, program and storage medium

A computer-implemented method for evaluating aspects of a document, the method comprising:

• receiving a document;
• receiving a system summary of the document, wherein the system summary of the document is a machine-generated summary of the document;
• receiving a plurality of reference summaries;
• dividing, based on a predetermined unit, the document into units of the document;
• dividing, based on the predetermined unit, the system summary of the document into units of the system summary;
• generating, based on the plurality of reference summaries and the document, a set of oracles associated with the plurality of reference summaries, wherein each oracle is a subset of units of the document that satisfy a text length restriction and maximize a score of an evaluation for the subset of the units of the document;
• determining, based on the generated set of oracles for each reference summary, a score of each unit included in the generated set of oracles;
• for each unit of the system summary, identifying a unit from the set of oracles;
• determining, based on the identified unit from the set of oracles, a score of each unit of the system summary;
• and
• determining, based on a set of the determined score of each unit of the system summary, a score for the system summary.

A system for evaluating aspects of a document, the system comprises:

• a processor;
• and
• a memory storing computer-executable instructions that when executed by the processor cause the system to:
• receive a document;
• receive a system summary of the document, wherein the system summary of the document is a machine-generated summary of the document;
• receive a plurality of reference summaries;
• divide, based on a predetermined unit, the document into units of the document;
• divide, based on the predetermined unit, the system summary of the document into units of the system summary;
• generate, based on the plurality of reference summaries and the document, a set of oracles associated with the plurality of reference summaries, wherein each oracle is a subset of units of the document that satisfy a text length restriction and maximize a score of an evaluation for the subset of the units of the document;
• determine, based on the generated set of oracles for each reference summary, a score of each unit included in the generated set of oracles;
• for each unit of the system summary, identify a unit from the set of oracles;
• determine, based on the identified unit from the set of oracles, a score of each unit of the system summary;
• and
• determine, based on a set of the determined score of each unit of the system summary, a score for the system summary.

A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:

• receive a document;
• receive a system summary of the document, wherein the system summary of the document is a machine-generated summary of the document;
• receive a plurality of reference summaries;
• divide, based on a predetermined unit, the document into units of the document;
• divide, based on the predetermined unit, the system summary of the document into units of the system summary;
• generate, based on the plurality of reference summaries and the document, a set of oracles associated with the plurality of reference summaries, wherein each oracle is a subset of units of the document that satisfy a text length restriction and maximize a score of an evaluation for the subset of the units of the document;
• determine, based on the generated set of oracles for each reference summary, a score of each unit included in the generated set of oracles;
• for each unit of the system summary, identify a unit from the set of oracles;
• determine, based on the identified unit from the set of oracles, a score of each unit of the system summary;
• and
• determine, based on a set of the determined score of each unit of the system summary, a score for the system summary.