US 10,380,490 B1Granted

Systems and methods for scoring story narrations

AssigneeEducational Testing Service

InventorsSwapna Somasundaran, Chong Min Lee, Martin Chodorow, Xinhao Wang

Life of this patent

Estimating the term…

Loading patent…

Abstract

Computer-based systems and methods are provided for generating a narrative computer scoring model for assessing story narratives. In one embodiment, supervised machine learning is used to generate the narrative computer scoring model. For example, a collection of training story narratives with assigned scores may be used to train the model. In one embodiment, each training story narrative is processed to extract features that signify content relevance, collocation of commonly used words, coherency, detailing, and expressions of sentiment. These features, as well as others, may be selectively used to train a narrative computer scoring model. Once trained, the model can be used to automatically evaluate story narratives and assign appropriate scores.

Claims25

Dependent claims

A computer-implemented method for generating a narrative computer scoring model for assessing story narratives, comprising:

retrieving, by a processing system, story narratives, wherein each story narrative is associated with a prompt that elicited the story narrative;
retrieving, by the processing system, reference corpora associated with the story narratives, wherein each story narrative's associated reference corpus includes content words relating to the associated prompt;
for each of the retrieved story narratives:
determining, by the processing system, overlaps between words in the story narrative and the content words in the associated reference corpus;
and
computing, by the processing system, at least one relevance feature associated with the story narrative using the determined overlaps;
retrieving, by the processing system, predetermined scores assigned to the story narratives by human raters;
generating, by the processing system, a narrative computer scoring model using the relevance features associated with the story narratives and the predetermined scores assigned to the story narratives;
and
storing, by the processing system, the narrative computer scoring model;
wherein the narrative computer scoring model is configured to output computer assessments of story narratives.

8 more elements

A computer-implemented system for generating a narrative computer scoring model for assessing story narratives, comprising:

one or more data processors;
and
a memory comprising instructions which when executed cause the one or more data processors to execute steps comprising:
retrieving story narratives, wherein each story narrative is associated with a prompt that elicited the story narrative;
retrieving reference corpora associated with the story narratives, wherein each story narrative's associated reference corpus includes content words relating to the associated prompt;
for each of the retrieved story narratives:
determining overlaps between words in the story narrative and the content words in the associated reference corpus;
and
computing at least one relevance feature associated with the story narrative using the determined overlaps;
retrieving predetermined scores assigned to the story narratives by human raters;
generating a narrative computer scoring model using the relevance features associated with the story narratives and the predetermined scores assigned to the story narratives;
and
storing the narrative computer scoring model;
wherein the narrative computer scoring model is configured to output computer assessments of story narratives.

11 more elements

A non-transitory computer-readable medium encoded with instructions for generating a narrative computer scoring model for assessing story narratives, the instructions being configured to cause a computer processing system to execute steps comprising:

retrieving story narratives, wherein each story narrative is associated with a prompt that elicited the story narrative;
retrieving reference corpora associated with the story narratives, wherein each story narrative's associated reference corpus includes content words relating to the associated prompt;
for each of the retrieved story narratives:
determining overlaps between words in the story narrative and the content words in the associated reference corpus;
and
computing at least one relevance feature associated with the story narrative using the determined overlaps;
retrieving predetermined scores assigned to the story narratives by human raters;
generating a narrative computer scoring model using the relevance features associated with the story narratives and the predetermined scores assigned to the story narratives;
and
storing the narrative computer scoring model;
wherein the narrative computer scoring model is configured to output computer assessments of story narratives.

8 more elements

Dependencies are inferred from the claim text.