Method of Model Scaling for an Automated Essay Scoring System
AssigneeEducational Testing Service
InventorYigal Attali
Life of this patent
Estimating the term…
Abstract
A method and system for customizing an automated essay scoring system are disclosed. Relative weights may be assigned to a plurality of features for a plurality of benchmark essay responses. Based on the relative weights, automated scores may be determined for one or more first essay responses to a first prompt. The automated scores may be scaled based on one or more scoring standards for the first prompt. A scaled automated score may be assigned for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses.
Claims21
A computer-implemented method of customizing an automated essay scoring system, the method comprising:
- assigning, using a processing system, relative weights to a plurality of features for a plurality of benchmark essay responses by using approximations calculated from performing regression analysis on essay responses responding to one or more essay prompts on other topics;
- determining, based on the relative weights, automated scores for one or more first essays responsive to a first prompt using the processing system;
- transforming the automated scores into scaled automated scores based on comparing statistical information for both human rater scores and the automated scores using the processing system, the scaled automated scores being representative of essay quality or writing competency;
- and
- assigning a scaled automated score for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses using the processing system;
- wherein transforming the automated scores into scaled automated scores comprises:
- receiving human rater scores for one or more benchmark essay responses;
- calculating human rater statistical information based on the human rater scores and automated statistical information based on the automated scores;
- and
- automatically adjusting the automated scores using the processing system based on the human rater statistical information and the automated statistical information.
A non-transitory computer-readable storage medium one or more programming instructions for customizing an automated essay scoring system, the programming instructions when executed causing a processing system to carry out steps comprising:
- assigning relative weights to a plurality of features for a plurality of benchmark essay responses by using approximations calculated from performing regression analysis on essay responses responding to one or more essay prompts on other topics;
- determining, based on the relative weights, automated scores for one or more first essays responsive to a first prompt;
- transforming the automated scores into scaled automated scores based on comparing statistical information for both human rater scores and the automated scores, the scaled automated scores being representative of essay quality or writing competency;
- and
- assigning a scaled automated score for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses;
- wherein transforming the automated scores into scaled automated scores comprises:
- receiving human rater scores for one or more benchmark essay responses;
- calculating human rater statistical information based on the human rater scores and automated statistical information based on the automated scores;
- and
- automatically adjusting the automated scores using a processor based on the human rater statistical information and the automated statistical information.
A system for customizing an automated essay scoring system, the system comprising:
- a processing system;
- a computer-readable memory comprising instructions for causing the processing system to execute steps comprising:
- assigning relative weights to a plurality of features for a plurality of benchmark essay responses by using approximations calculated from performing regression analysis on essay responses responding to one or more essay prompts on other topics;
- determining, based on the relative weights, automated scores for one or more first essays responsive to a first prompt;
- transforming the automated scores into scaled automated scores based on comparing statistical information for both human rater scores and the automated scores, the scaled automated scores being representative of essay quality or writing competency;
- and
- assigning a scaled automated score for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses;
- wherein transforming the automated scores into scaled automated scores comprises:
- receiving human rater scores for one or more benchmark essay responses;
- calculating human rater statistical information based on the human rater scores and automated statistical information based on the automated scores;
- and
- automatically adjusting the automated scores using the processor based on the human rater statistical information and the automated statistical information.
Dependencies are inferred from the claim text.