System and Method for Automated Scoring of a Summary-Writing Task

A system for measuring a user's comprehension of subject matter of a text, the system comprising:

• a processing system;
• and
• computer-readable memory in communication with the processing system encoded with instructions for commanding the processing system to execute steps comprising:
• receiving a summary generated by the user, the summary being a constructed response that summarizes a text;
• parsing the summary with the processing system to identify a number of sentences contained in the summary and to identify in the summary a plurality of multi-word sequences;
• processing the summary and a reference summary with the processing system to determine a first numerical measure indicative of a similarity between the summary and a reference summary, the reference summary having been designated as representative of the subject matter of the text;
• processing the summary with the processing system to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text;
• processing the summary and the text with the processing system to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text;
• and
• applying a numerical model to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text, the numerical model including
• a first variable and an associated first weighting factor, the first variable receiving a value of the first numerical measure,
• a second variable and an associated second weighting factor, the first variable receiving a value of the second numerical measure, and
• a third variable and an associated third weighting factor, the third variable receiving a value of the third numerical measure.

A non-transitory computer-readable storage medium for measuring a user's comprehension of subject matter of a text, the computer-readable storage medium comprising computer executable instructions which, when executed, cause a processing system to execute steps comprising:

• receiving a summary generated by the user, the summary being a constructed response that summarizes a text;
• parsing the summary with the processing system to identify a number of sentences contained in the summary and to identify in the summary a plurality of multi-word sequences;
• processing the summary and a reference summary with the processing system to determine a first numerical measure indicative of a similarity between the summary and a reference summary, the reference summary having been designated as representative of the subject matter of the text;
• processing the summary with the processing system to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text;
• processing the summary and the text with the processing system to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text;
• and
• applying a numerical model to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text, the numerical model including
• a first variable and an associated first weighting factor, the first variable receiving a value of the first numerical measure,
• a second variable and an associated second weighting factor, the first variable receiving a value of the second numerical measure, and
• a third variable and an associated third weighting factor, the third variable receiving a value of the third numerical measure.

A computer-implemented method of constructing a numerical model to measure a user's comprehension of subject matter of a text as presented in a summary of the text constructed by a user, the method comprising:

• specifying a numerical model associated with a given text, the numerical model comprising
• a first variable and an associated first weighting factor, the first variable indicative of a similarity between a summary of the given text constructed by a user and a given reference summary,
• a second variable and an associated second weighting factor, the second variable indicative of a degree to which a single sentence of the summary summarizes an entirety of the given text, and
• a third variable and an associated third weighting factor, the third variable indicative of a degree of copying in the summary of multi-word sequences present in the given text;
• receiving a plurality of reference summaries for the given text, each reference summary having been given a reference score, the reference summaries spanning a range of reference scores reflecting varying degrees of comprehension of the subject matter of the given text, the reference summaries having been accepted as usable for training the numerical model;
• training the numerical model with a processing system using the reference summaries and the given reference scores to determine values for each of the first, second and third weighting factors;
• and
• configuring the numerical model with the determined values of the first, second and third weighting factors to receive a first numerical measure, a second numerical measure and a third numerical measure for the first variable, second variable and third variable, respectively, of an actual summary to be scored so as to generate a score for the actual summary that is indicative of the user's comprehension of the subject matter of the text as presented in a summary of the text.

A system for constructing a numerical model to measure a user's comprehension of subject matter of a text as presented in a summary of the text constructed by a user, the system comprising:

• a processing system;
• and
• computer-readable memory in communication with the processing system encoded with instructions for commanding the processing system to execute steps comprising:
• specifying a numerical model associated with a given text, the numerical model comprising
• a first variable and an associated first weighting factor, the first variable indicative of a similarity between a summary of the given text constructed by a user and a given reference summary,
• a second variable and an associated second weighting factor, the second variable indicative of a degree to which a single sentence of the summary summarizes an entirety of the given text, and
• a third variable and an associated third weighting factor, the third variable indicative of a degree of copying in the summary of multi-word sequences present in the given text;
• receiving a plurality of reference summaries for the given text, each reference summary having been given a reference score, the reference summaries spanning a range of reference scores reflecting varying degrees of comprehension of the subject matter of the given text, the reference summaries having been accepted as usable for training the numerical model;
• training the numerical model with the processing system using the reference summaries and the given reference scores to determine values for each of the first, second and third weighting factors;
• and
• configuring the numerical model with the determined values of the first, second and third weighting factors to receive a first numerical measure, a second numerical measure and a third numerical measure for the first variable, second variable and third variable, respectively, of an actual summary to be scored so as to generate a score for the actual summary that is indicative of the user's comprehension of the subject matter of the text as presented in a summary of the text.

A non-transitory computer-readable storage medium for constructing a numerical model to measure a user's comprehension of subject matter of a text as presented in a summary of the text constructed by a user, the computer-readable storage medium comprising computer executable instructions which, when executed, cause a processing system to execute steps comprising:

• specifying a numerical model associated with a given text, the numerical model comprising
• a first variable and an associated first weighting factor, the first variable indicative of a similarity between a summary of the given text constructed by a user and a given reference summary,
• a second variable and an associated second weighting factor, the second variable indicative of a degree to which a single sentence of the summary summarizes an entirety of the given text, and
• a third variable and an associated third weighting factor, the third variable indicative of a degree of copying in the summary of multi-word sequences present in the given text;
• receiving a plurality of reference summaries for the given text, each reference summary having been given a reference score, the reference summaries spanning a range of reference scores reflecting varying degrees of comprehension of the subject matter of the given text, the reference summaries having been accepted as usable for training the numerical model;
• training the numerical model with the processing system using the reference summaries and the given reference scores to determine values for each of the first, second and third weighting factors;
• and
• configuring the numerical model with the determined values of the first, second and third weighting factors to receive a first numerical measure, a second numerical measure and a third numerical measure for the first variable, second variable and third variable, respectively, of an actual summary to be scored so as to generate a score for the actual summary that is indicative of the user's comprehension of the subject matter of the text as presented in a summary of the text.