System and method for automated scoring of texual responses to picture-based items

A system for constructing a model to measure a user's English language proficiency as presented in a constructed response generated by the user, the constructed response being based on a given picture, the system comprising:

• a processing system;
• and
• computer-readable memory in communication with the processing system encoded with instructions for commanding the processing system to execute steps comprising:
• specifying a model associated with the given picture, the model comprising
• a first variable and an associated first weighting factor, the first variable indicative of a presence of one or more grammar errors included in the constructed response,
• a second variable and an associated second weighting factor, the second variable indicative of a degree to which the constructed response describes a subject matter of the given picture as determined based on a reference corpus designated as representative of the subject matter, and
• a third variable and an associated third weighting factor, the third variable indicative of a degree of awkward word usage in the constructed response as determined based on an n-gram dataset with entries including an English word n-gram and an associated statistical association score;
• receiving a plurality of reference responses for the given picture, each reference response having been given a reference score;
• training the model with the processing system using the reference responses and the given reference scores to determine values for each of the first, second and third weighting factors;
• and
• configuring the model with the determined values of the first, second and third weighting factors to receive a first numerical measure, a second numerical measure, and a third numerical measure for the first variable, second variable, and third variable, respectively, of an actual constructed response to be scored so as to generate a score for the actual constructed response that is indicative of the user's English language proficiency as presented in the actual constructed response generated by the user.

A non-transitory computer-readable storage medium for constructing a model to measure a user's English language proficiency as presented in a constructed response generated by the user, the constructed response being based on a given picture, the computer-readable storage medium comprising computer executable instructions which, when executed, cause a processing system to execute steps comprising:

• specifying a model associated with the given picture, the model comprising
• a first variable and an associated first weighting factor, the first variable indicative of a presence of one or more grammar errors included in the constructed response,
• a second variable and an associated second weighting factor, the second variable indicative of a degree to which the constructed response describes a subject matter of the given picture as determined based on a reference corpus designated as representative of the subject matter, and
• a third variable and an associated third weighting factor, the third variable indicative of a degree of awkward word usage in the constructed response as determined based on an n-gram dataset with entries including an English word n-gram and an associated statistical association score;
• receiving a plurality of reference responses for the given picture, each reference response having been given a reference score;
• training the model with the processing system using the reference responses and the given reference scores to determine values for each of the first, second and third weighting factors;
• and
• configuring the model with the determined values of the first, second and third weighting factors to receive a first numerical measure, a second numerical measure, and a third numerical measure for the first variable, second variable, and third variable, respectively, of an actual constructed response to be scored so as to generate a score for the actual constructed response that is indicative of the user's English language proficiency as presented in the actual constructed response generated by the user.