Computer-implemented method of administering and scoring integrated reasoning question formats

An automated system for presenting and scoring items which are presented in a question format that includes (i) a graphic representation of data or information, (ii) one or more statements which relate to the graphical representation, and (iii) a plurality of response opportunities in the form of pull-down menus embedded within the one or more statements, each of the pull-down menus presenting a plurality of responses to a response opportunity, each response opportunity being independent of each other, the system comprising:

• (a) a processor-controlled user interface display screen for presenting items in the question format;
• (b) a response engine that receives the response for each response opportunity of each item;
• and
• (c) a scoring engine in communication with the response engine which receives the plurality of responses and scores the plurality of responses of each item by:
• (i) determining whether the response selected for each response opportunity is correct, and
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated method for presenting and scoring items which are presented in a question format that includes (i) sortable table of data and explanatory table information, (ii) a plurality of statements related to the sortable table of data and explanatory table information, and (iii) a dichotomous-choice response opportunity for each of statements, each response opportunity being independent of each other, the method comprising for each item:

• (a) presenting on a processor-controlled user interface display screen an item in the question format;
• (b) receiving at a response engine the response for each of the dichotomous-choice response opportunities;
• and
• (c) scoring the plurality of responses in a scoring engine which is in communication with the response engine and which receives the plurality of responses, the plurality of responses being scored by:
• (i) determining whether the response selected for each response opportunity is correct,
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated system for presenting and scoring items which are presented in a question format that includes (i) sortable table of data and explanatory table information, (ii) a plurality of statements related to the sortable table of data and explanatory table information, and (iii) a dichotomous-choice response opportunity for each of statements, each response opportunity being independent of each other, the system comprising:

• (a) a processor-controlled user interface display screen for presenting items in the question format;
• (b) a response engine that receives the response for each of the dichotomous-choice response opportunities of each item;
• and
• (c) a scoring engine in communication with the response engine which receives the plurality of responses and scores the plurality of responses of each item by:
• (i) determining whether the response selected for each response opportunity is correct,
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated method for presenting and scoring items which are presented in a question format that includes (i) stimulus text, (ii) a stem, and (iii) a multiple column response table including a column of response options and a plurality of multiple choice columns, each multiple choice column providing a respective response opportunity with a header related to the stem, each response opportunity being independent of each other, wherein the same column of response options are used for each of the multiple choice columns, and wherein respondents select one choice for each of the multiple choice columns as their response, the method comprising for each item:

• (a) presenting on a processor-controlled user interface display screen an item in the question format;
• (b) receiving at a response engine the response for each response opportunity presented on the user-interface display screen;
• and
• (c) scoring the plurality of responses in a scoring engine which is in communication with the response engine and which receives the plurality of responses, the plurality of responses being scored by:
• (i) determining whether the response selected for each response opportunity is correct,
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated system for presenting and scoring items which are presented in a question format that includes (i) stimulus text, (ii) a stem, and (iii) a multiple column response table including a column of response options and a plurality of multiple choice columns, each multiple choice column providing a respective response opportunity with a header related to the stem, each response opportunity being independent of each other, wherein the same column of response options are used for each of the multiple choice columns, and wherein respondents select one choice for each of the multiple choice columns as their response, the system comprising:

• (a) a processor-controlled user interface display screen for presenting items in the question format;
• (b) a response engine that receives the response for each response opportunity of each item;
• and
• (c) a scoring engine in communication with the response engine which receives the plurality of responses and scores the plurality of responses of each item by:
• (i) determining whether the response selected for each response opportunity is correct,
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated method for presenting and scoring a plurality of items in an item set which are presented in a question format that includes (i) stimulus information presented in multiple tabs, (ii) a plurality of statements related to the stimulus information, and (iii) a plurality of response opportunities, including a dichotomous-choice response opportunity for some of the statements, each response opportunity being independent of each other, wherein the plurality of items in the item set use the same stimulus information, the method comprising for each item in the item set having a dichotomous-choice response opportunity:

• (a) presenting on a processor-controlled user interface display screen an item in the question format;
• (b) receiving at a response engine the response for each of the dichotomous-choice response opportunities;
• (c) scoring the plurality of responses in a scoring engine which is in communication with the response engine and which receives the plurality of responses, the plurality of responses being scored by:
• (i) determining whether the response selected for each response opportunity is correct,
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated system for presenting and scoring a plurality of items in an item set which are presented in a question format that includes (i) stimulus information presented in multiple tabs, (ii) a plurality of statements related to the stimulus information;

• (iii) a plurality of response opportunities, including a dichotomous-choice response opportunity for some of the statements, each response opportunity being independent of each other, wherein the plurality of items in the item set use the same stimulus information, the system comprising:
• (a) a processor-controlled user interface display screen for presenting items in the question format having a dichotomous-choice response opportunity;
• (b) a response engine that receives the response for each response opportunity of each item in the item set having a dichotomous-choice response opportunity;
• and
• (c) a scoring engine in communication with the response engine which receives the plurality of responses and scores the plurality of responses of each item in the item set having a dichotomous-choice response opportunity by:
• (i) determining whether the response selected for each response opportunity is correct,
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated method for presenting and scoring items which are presented in integrated reasoning question formats that include:

• a first question format having (i) a graphic representation of data or information, (ii) one or more statements which relate to the graphical representation, and (iii) a plurality of response opportunities in the form of pull-down menus embedded within the one or more statements, each of the pull-down menus presenting a plurality of responses to a response opportunity;
• a second question format having (i) sortable table of data and explanatory table information, (ii) a plurality of statements related to the sortable table of data and explanatory table information, and (iii) a dichotomous-choice response opportunity for each of statements;
• a third question format having (i) stimulus text, (ii) a stem, and (iii) a multiple column response table including a column of response options and a plurality of multiple choice columns, each multiple choice column providing a respective response opportunity with a header related to the stem, wherein the same column of response options are used for each of the multiple choice columns, and wherein respondents select one choice for each of the multiple choice columns as their response;
• and
• a fourth question format having a plurality of items in an item set which are presented in a question format that includes (i) stimulus information presented in multiple tabs, (ii) a plurality of statements related to the stimulus information, and (iii) a plurality of response opportunities, including a dichotomous-choice response opportunity for some of the statements, wherein the plurality of items in the item set use the same stimulus information,
• each response opportunity being independent of each other, the method comprising for each item or for each item in the item set having a dichotomous-choice response opportunity:
• (a) presenting on a processor-controlled user interface display screen an item in the question format;
• (b) receiving at a response engine the response for each response opportunity presented on the user interface display screen;
• and
• (c) scoring the plurality of responses in a scoring engine which is in communication with the response engine and which receives the plurality of responses, the plurality of responses being scored by:
• (i) determining whether the response selected for each response opportunity is correct, and
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.

An automated system for presenting and scoring items which are presented in integrated reasoning question formats that include:

• a first question format having (i) a graphic representation of data or information, (ii) one or more statements which relate to the graphical representation, and (iii) a plurality of response opportunities in the form of pull-down menus embedded within the one or more statements, each of the pull-down menus presenting a plurality of responses to a response opportunity;
• a second question format having (i) sortable table of data and explanatory table information, (ii) a plurality of statements related to the sortable table of data and explanatory table information, and (iii) a dichotomous-choice response opportunity for each of statements;
• a third question format having (i) stimulus text, (ii) a stem, and (iii) a multiple column response table including a column of response options and a plurality of multiple choice columns, each multiple choice column providing a respective response opportunity with a header related to the stem, wherein the same column of response options are used for each of the multiple choice columns, and wherein respondents select one choice for each of the multiple choice columns as their response;
• and
• a fourth question format having a plurality of items in an item set which are presented in a question format that includes (i) stimulus information presented in multiple tabs, (ii) a plurality of statements related to the stimulus information, and (iii) a plurality of response opportunities, including a dichotomous-choice response opportunity for some of the statements, wherein the plurality of items in the item set use the same stimulus information,
• each response opportunity being independent of each other, the system comprising:
• (a) a processor-controlled user interface display screen for presenting items in the question format;
• (b) a response engine that receives the response for each response opportunity of each item;
• and
• (c) a scoring engine in communication with the response engine which receives the plurality of responses and scores the plurality of responses of each item by:
• (i) determining whether the response selected for each response opportunity is correct, and
• (ii) assigning credit for the item only if each of the responses for each of the response opportunities is correct, and assigning no credit for the item if at least one of the responses for each of the response opportunities is incorrect.