Systems and methods for evaluating susceptibility of automated scoring engines to gaming behavior

A computer-implemented system for reducing an inflation of a score for a constructed response to a prompt, the system comprising:

• a computer-readable medium encoded with a plurality of constructed responses to a prompt, the plurality of constructed responses associated with a plurality of test takers;
• a computer-implemented transformation engine configured to:
• select one or more predetermined transformations from a set of transformations to be performed on each of the plurality of constructed responses, the one or more predetermined transformations corresponding to one or more gaming strategies that adjust constructed responses to produce an inflated score;
• generate a first set of transformed responses based on the selected one or more predetermined transformations and the plurality of constructed responses;
• a computer-implemented automated scoring engine configured to:
• receive the plurality of constructed responses and to generate a first set of scores for the plurality of constructed responses;
• receive the first set of transformed responses and to generate a second set of scores for the first set of transformed responses;
• a computer-implemented score comparison module configured to:
• determine a gaming susceptibility metric based on the first set of scores and the second set of scores, the gaming susceptibility metric indicating that the computer-implemented automated scoring engine is susceptible to the one or more gaming strategies;
• adjust the computer-implemented automated scoring engine based on the gaming susceptibility metric;
• receive a live constructed response and provide the live constructed response to the adjusted scoring engine;
• detect that the live constructed response exhibits at least one of the one or more gaming strategies;
• and
• lower the score for the live constructed response using the adjusted computer-implemented automated scoring engine.

A non-transitory computer-readable medium encoded with instructions for commanding one or more data processors to perform a method for reducing an inflation of a score for a constructed response to a prompt, the method comprising:

• receiving a plurality of constructed responses to a prompt, the plurality of constructed responses associated with a plurality of test takers;
• generating, using a computer-implemented automated scoring engine, a first set of scores for the plurality of constructed responses;
• selecting one or more predetermined transformations from a set of transformations to be performed on each of the plurality of constructed responses, the one or more predetermined transformations corresponding to one or more gaming strategies that adjust constructed responses to produce an inflated score;
• generating a first set of transformed responses based on the selected one or more predetermined transformations and the plurality of constructed responses;
• generating using the computer-implemented automated scoring engine, a second set of scores for the first set of transformed responses;
• determining a gaming susceptibility metric based on the first set of scores and the second set of scores, the gaming susceptibility metric indicating that the computer-implemented automated scoring engine is susceptible to the one or more gaming strategies;
• adjusting the computer-implemented automated scoring engine based on the gaming susceptibility metric;
• receiving a live constructed response and providing the live constructed response to the adjusted scoring engine;
• detecting that the live constructed response exhibits at least one of the one or more gaming strategies;
• and
• lowering the score for the live constructed response using the adjusted computer-implemented automated scoring engine.