# US 2020/0401797 A1 · Summary evaluation device, method, program and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401797 A1
- Document ID: US20200401797A1
- Status: Published application
- Kind: A1
- Application number: 16/975,372
- Filed: 2019-02-14
- Published: 2020-12-24
- Earliest claimed: 2018-02-26
- Assignees: Nippon Telegraph and Telephone Corporation
- Applicants: Nippon Telegraph and Telephone Corporation
- Inventors: Tsutomu HIRAO; Masaaki NAGATA
- Web page: https://patentgrep.com/patent/US20200401797A1

## CPC

- G06F 40/279: Recognition of textual entities
- G06F 16/345: {Summarisation for human users}
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/2163: {Partitioning the feature space}
- G06F 18/2185: {the supervisor being an automated module, e.g. intelligent oracle}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/30: Semantic analysis
- G06F 40/56: Natural language generation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system summary can be evaluated accurately with consideration given to the meaning of each unit. A unit division portion divides a document to be summarized and a system summary into units. For each reference summary, an oracle generation portion generates an oracle, which is a subset of units of the document to be summarized that satisfy a length requirement and maximize a score of an evaluation function for the subset of units with respect to the reference summary. An oracle unit score determination portion determines the score of each unit included in a set of the oracles based on the generated oracles. For each unit of the system summary, a system unit score determination portion obtains a corresponding unit from among the units included in the set of the oracles and determines the score of the unit of the system summary. An evaluation score determination portion determines the score of the system summary based on the scores of the units of the system summary.

## Claims

### Claim at position 1

-7. (canceled)

### Claim 8

A computer-implemented method for evaluating aspects of a document, the method comprising:
receiving a document;
receiving a system summary of the document, wherein the system summary of the document is a machine-generated summary of the document;
receiving a plurality of reference summaries;
dividing, based on a predetermined unit, the document into units of the document;
dividing, based on the predetermined unit, the system summary of the document into units of the system summary;
generating, based on the plurality of reference summaries and the document, a set of oracles associated with the plurality of reference summaries, wherein each oracle is a subset of units of the document that satisfy a text length restriction and maximize a score of an evaluation for the subset of the units of the document;
determining, based on the generated set of oracles for each reference summary, a score of each unit included in the generated set of oracles;
for each unit of the system summary, identifying a unit from the set of oracles;
determining, based on the identified unit from the set of oracles, a score of each unit of the system summary; and
determining, based on a set of the determined score of each unit of the system summary, a score for the system summary.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the oracle is a set of units that maximizes at least one of automatic summary evaluation processes as the evaluation, the evaluation include one of: the ROUGE, and Basic Elements.

### Claim 10 (depends on 8)

The computer-implemented method of claim 8, wherein the predetermined unit comprises one or more of a sentence and a phrase in the document.

### Claim 11 (depends on 8)

The computer-implemented method of claim 8, the method further comprising:
for each unit of the system summary, allocating a corresponding unit in the set of oracles, wherein the corresponding unit is a distinct unit among the units in the set of oracles.

### Claim 12 (depends on 8)

The computer-implemented method of claim 8, the method further comprising:
for each unit of the system summary, receiving, based on a degree of similarity between the unit and respective units in the set of oracles, a corresponding unit in the units in the set of oracles, wherein the degree of similarity is based on a partial character string that is common between the unit and the respective units.

### Claim 13 (depends on 8)

The computer-implemented method of claim 8, the method further comprising:
selecting, from the units in the set of oracles, at least one combination of units satisfying the length restriction;
generating the maximum score of the set of oracles based on the selected at least one combination of units; and
normalizing the score of the unit of the system summary based on the generated maximized score for determining the score for the system summary.

### Claim 14 (depends on 8)

The computer-implemented method of claim 8, wherein:
when the generated units of the documents and the generated units of the system summary each represents a continuous series of units in texts based on one of:
a sentence using one or more punctuation marks and sentence boundaries, or
a phrase based on phrase boundary identifications.

### Claim 15

A system for evaluating aspects of a document, the system comprises:
a processor; and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
receive a document;
receive a system summary of the document, wherein the system summary of the document is a machine-generated summary of the document;
receive a plurality of reference summaries;
divide, based on a predetermined unit, the document into units of the document;
divide, based on the predetermined unit, the system summary of the document into units of the system summary;
generate, based on the plurality of reference summaries and the document, a set of oracles associated with the plurality of reference summaries, wherein each oracle is a subset of units of the document that satisfy a text length restriction and maximize a score of an evaluation for the subset of the units of the document;
determine, based on the generated set of oracles for each reference summary, a score of each unit included in the generated set of oracles;
for each unit of the system summary, identify a unit from the set of oracles;
determine, based on the identified unit from the set of oracles, a score of each unit of the system summary; and
determine, based on a set of the determined score of each unit of the system summary, a score for the system summary.

### Claim 16 (depends on 15)

The system of claim 15, wherein the oracle is a set of units that maximizes at least one of automatic summary evaluation processes as the evaluation, the evaluation include one of: the ROUGE, and Basic Elements.

### Claim 17 (depends on 15)

The system of claim 15, wherein the predetermined unit comprises one or more of a sentence and a phrase in the document.

### Claim 18 (depends on 15)

The system of claim 15, the computer-executable instructions when executed further causing the system to:
for each unit of the system summary, allocate a corresponding unit in the set of oracles, wherein the corresponding unit is a distinct unit among the units in the set of oracles.

### Claim 19 (depends on 15)

The system of claim 15, the computer-executable instructions when executed further causing the system to:
for each unit of the system summary, receive, based on a degree of similarity between the unit and respective units in the set of oracles, a corresponding unit in the units in the set of oracles, wherein the degree of similarity is based on a partial character string that is common between the unit and the respective units.

### Claim 20 (depends on 15)

The system of claim 15, the computer-executable instructions when executed further causing the system to:
select, from the units in the set of oracles, at least one combination of units satisfying the length restriction;
generate the maximum score of the set of oracles based on the selected at least one combination of units; and
normalize the score of the unit of the system summary based on the generated maximized score for determining the score for the system summary.

### Claim 21 (depends on 15)

The system of claim 15, wherein:
when the generated units of the documents and the generated units of the system summary each represents a continuous series of units in texts based on one of:
a sentence using one or more punctuation marks and sentence boundaries, or
a phrase based on phrase boundary identifications.

### Claim 22

A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:
receive a document;
receive a system summary of the document, wherein the system summary of the document is a machine-generated summary of the document;
receive a plurality of reference summaries;
divide, based on a predetermined unit, the document into units of the document;
divide, based on the predetermined unit, the system summary of the document into units of the system summary;
generate, based on the plurality of reference summaries and the document, a set of oracles associated with the plurality of reference summaries, wherein each oracle is a subset of units of the document that satisfy a text length restriction and maximize a score of an evaluation for the subset of the units of the document;
determine, based on the generated set of oracles for each reference summary, a score of each unit included in the generated set of oracles;
for each unit of the system summary, identify a unit from the set of oracles;
determine, based on the identified unit from the set of oracles, a score of each unit of the system summary; and
determine, based on a set of the determined score of each unit of the system summary, a score for the system summary.

### Claim 23 (depends on 22)

The computer-readable non-transitory recording medium of claim 22,
wherein the oracle is a set of units that maximizes at least one of automatic summary evaluation processes as the evaluation, the evaluation include one of: the ROUGE, and Basic Elements.

### Claim 24 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, wherein the predetermined unit comprises one or more of a sentence and a phrase in the document.

### Claim 25 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, the computer-executable instructions when executed further causing the system to:
for each unit of the system summary, allocate a corresponding unit in the set of oracles, wherein the corresponding unit is a distinct unit among the units in the set of oracles.

### Claim 26 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, the computer-executable instructions when executed further causing the system to:
for each unit of the system summary, receive, based on a degree of similarity between the unit and respective units in the set of oracles, a corresponding unit in the units in the set of oracles, wherein the degree of similarity is based on a partial character string that is common between the unit and the respective units.

### Claim 27 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, the computer-executable instructions when executed further causing the system to:
select, from the units in the set of oracles, at least one combination of units satisfying the length restriction;
generate the maximum score of the set of oracles based on the selected at least one combination of units; and
normalize the score of the unit of the system summary based on the generated maximized score for determining the score for the system summary.