Systems and Methods for Generating Automated Evaluation Models

A computer-implemented method of identifying stable features, comprising:

• accessing a plurality of training responses with a processing system, the training responses being constructed responses;
• analyzing the plurality of training responses with the processing system to derive values of multiple candidate features of the training responses;
• computing correlation measures with the processing system based on the derived values of the multiple candidate features and a portfolio score for each individual associated with the plurality of training responses, wherein the portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual, and wherein the multiple writing samples constructed by each individual do not include any of the plurality of training responses and the portfolio score for each individual is not derived from any of the plurality of training responses;
• comparing the correlation measures to at least one predetermined selection criterion;
• and
• selecting a set of features from the multiple candidate features based on said step of comparing.

A non-transitory computer-readable medium encoded with instructions for causing a processing system to execute steps for calibrating an assessment model for assessing responses, comprising:

• accessing a plurality of training responses with a processing system for training an assessment model for assessing responses, the training responses being constructed responses;
• analyzing the plurality of training responses with the processing system to derive values of multiple features of the training responses, the multiple features corresponding to variables of the assessment model;
• training the assessment model with the processing system based on the values of the multiple features of the training responses and a portfolio score for each individual associated with the plurality of training responses utilized in the training, wherein the portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual, and wherein the multiple writing samples constructed by each individual do not include any of the plurality of training responses and the portfolio score for each individual is not derived from any of the plurality of training responses;
• determining, based on said training, a weight for each of the multiple features;
• and
• calibrating the assessment model to include the weights for at least some of the features such that the assessment model is configured to generate scores for responses.

A non-transitory computer-readable medium encoded with instructions for causing a processing system to execute steps for identifying stable features, comprising:

• accessing a plurality of training responses with a processing system, the training responses being constructed responses;
• analyzing the plurality of training responses with the processing system to derive values of multiple candidate features of the training responses;
• computing correlation measures with the processing system based on the derived values of the multiple candidate features and a portfolio score for each individual associated with the plurality of training responses, wherein the portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual, and wherein the multiple writing samples constructed by each individual do not include any of the plurality of training responses and the portfolio score for each individual is not derived from any of the plurality of training responses;
• comparing the correlation measures to at least one predetermined selection criterion;
• and
• selecting a set of features from the multiple candidate features based on said step of comparing.

A system for calibrating an assessment model for assessing responses, comprising:

• a processing system;
• and
• a memory coupled to the processing system, wherein the processing system is configured to execute steps, comprising:
• accessing a plurality of training responses with a processing system for training an assessment model for assessing responses, the training responses being constructed responses;
• analyzing the plurality of training responses with the processing system to derive values of multiple features of the training responses, the multiple features corresponding to variables of the assessment model;
• training the assessment model with the processing system based on the values of the multiple features of the training responses and a portfolio score for each individual associated with the plurality of training responses utilized in the training, wherein the portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual, and wherein the multiple writing samples constructed by each individual do not include any of the plurality of training responses and the portfolio score for each individual is not derived from any of the plurality of training responses;
• determining, based on said training, a weight for each of the multiple features;
• and
• calibrating the assessment model to include the weights for at least some of the features such that the assessment model is configured to generate scores for responses.

A system for calibrating an assessment model for identifying stable features, comprising:

• a processing system;
• and
• a memory coupled to the processing system, wherein the processing system is configured to execute steps, comprising:
• accessing a plurality of training responses with a processing system, the training responses being constructed responses;
• analyzing the plurality of training responses with the processing system to derive values of multiple candidate features of the training responses;
• computing correlation measures with the processing system based on the derived values of the multiple candidate features and a portfolio score for each individual associated with the plurality of training responses, wherein the portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual, and wherein the multiple writing samples constructed by each individual do not include any of the plurality of training responses and the portfolio score for each individual is not derived from any of the plurality of training responses;
• comparing the correlation measures to at least one predetermined selection criterion;
• and
• selecting a set of features from the multiple candidate features based on said step of comparing.