Summary evaluation device, method, program, and storage medium

A computer-implemented method for evaluating aspects of a document, the method comprising:

• receiving a plurality of reference summaries of the document;
• receiving a system summary of the document, wherein the system summary is a machine-generated summary of the document;
• generating at least a first set of tuples for one of the plurality of reference summaries and at least a second set of tuples for the system summary, wherein each tuple comprises:
• a head word,
• a modifier word having a dependency relation with the head word, and
• a label indicating the dependency relation based on one or more reference summaries of the plurality of reference summaries and the system summary;
• for each of one or more tuples of at least the first and the second sets of tuples, replacing the head word with a first class of words and the modifier word with a second class of words;
• determining a score of the system summary for evaluating the system summary of the document based at least on the first set of tuples of the plurality of reference summaries with the replaced first class of words and the second set of tuples of the system summary with the replaced second class of words;
• and
• providing the score.

A system for evaluating aspects of a document, the system comprises:

• a processor;
• and
• a memory storing computer-executable instructions that when executed by the processor cause the system to:
• receive a plurality of reference summaries of the document;
• receive a system summary of the document, wherein the system summary is a machine-generated summary of the document;
• generate at least a first set of tuples for one of the plurality of reference summaries and at least a second set of tuples for the system summary, wherein each tuple comprises:
• a head word,
• a modifier word having a dependency relation with the head word, and
• a label indicating the dependency relation based on one or more reference summaries of the plurality of reference summaries and the system summary;
• for each of one or more tuples of at least the first and the second sets of tuples, replace the head word with a first class of words and the modifier word with a second class of words;
• determine a score of the system summary for evaluating the system summary of the document based at least on the first set of tuples of the plurality of reference summaries with the replaced first class of words and the second set of tuples of the system summary with the replaced second class of words;
• and
• provide the score.

A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:

• receive a plurality of reference summaries of the document;
• receive a system summary of the document, wherein the system summary is a machine-generated summary of the document;
• generate at least a first set of tuples for one of the plurality of reference summaries and at least a second set of tuples for the system summary, wherein each tuple comprises:
• a head word,
• a modifier word having a dependency relation with the head word, and
• a label indicating the dependency relation based on one or more reference summaries of the plurality of reference summaries and the system summary;
• for each of one or more tuples of at least the first and the second sets of tuples, replace the head word with a first class of words and the modifier word with a second class of words;
• determine a score of the system summary for evaluating the system summary of the document based at least on the first set of tuples of the plurality of reference summaries with the replaced first class of words and the second set of tuples of the system summary with the replaced second class of words;
• and
• provide the score.