Collaborative and quality control scoring system and method

A system for evaluating the performance of test resolvers by routing quality-control questions for scoring to test resolvers, the quality-control questions comprising a test answer with a score known to quality-control personnel, the system comprising:

• a) transmit means for electronically transmitting to a test resolver an unidentified quality-control question comprising a test answer with a score known to quality-control personnel;
• b) receive means for electronically receiving from the test resolver a score for the quality-control question;
• and
• c) processing means for electronically comparing the score known to the quality-control personnel and the resolver score and for reporting a discrepancy between the known score and the resolver score.

A system for electronically providing performance feedback based upon a comparison of a test resolver's performance to predetermined performance criteria, the system comprising:

• a) receive means for electronically receiving information related to a first performance level from a first resolver and a predetermined performance level;
• b) comparison means for electronically comparing the first performance level and the predetermined performance level;
• and
• c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first performance level and the predetermined performance level.

A system for electronically providing performance feedback based upon a comparison of a test resolver's performance to predetermined performance criteria, the system comprising:

• a) receive means for electronically receiving information related to a first test-score validity level from a first resolver and a predetermined validity level;
• b) comparison means for electronically comparing the first validity level and the predetermined validity level;
• and
• c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first validity level and the predetermined validity level.

A system for electronically providing performance feedback based upon a comparison of a test resolver's performance to predetermined performance criteria, the system comprising:

• a) receive means for electronically receiving information related to a first reliability level from a first resolver and a predetermined reliability level;
• b) comparison means for electronically comparing the first reliability level and the predetermined reliability level;
• and
• c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first reliability level and the predetermined reliability level.

A system for electronically providing performance feedback based upon a comparison of a test resolver's performance to predetermined performance criteria, the system comprising:

• a) receive means for electronically receiving information related to a first speed level from a first resolver and a predetermined speed level;
• b) comparison means for electronically comparing the first speed level and the predetermined speed level;
• and
• c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first speed level and the predetermined speed level.

A system for increasing test resolver efficiency by monitoring test resolver performance and recommending break times to the test resolvers, the system comprising:

• a) receive means for electronically receiving at predetermined past intervals in time first performance levels from a test resolver and for electronically receiving a second predetermined performance level;
• b) comparison means for electronically comparing the second predetermined performance level and the first performance level;
• and
• c) recommendation means for electronically detecting a discrepancy based on predefined criteria between the second predetermined performance level and the first performance level as indicated by the comparison means and for signalling the test resolver to halt test scoring for a break time.

A system for increasing test resolver efficiency by monitoring test resolver performance and recommending break times to the test resolvers, the system comprising:

• a) receive means for electronically receiving at predetermined past intervals in time first test scoring rates from a test resolver and for electronically receiving a second present scoring rate from the test resolver;
• b) comparison means for electronically comparing the second scoring rate and the first scoring rates;
• and
• c) recommendation means for electronically detecting a discrepancy based on predefined criteria between the second scoring rate and the first scoring rates as indicated by the comparison means and for signalling the test resolver to halt test scoring for a break time.

A system for preventing bias in the scoring of answers to tests taken by persons in different particular classifications, the system comprising:

• a) receive means for electronically receiving a plurality of answers to test questions, the test answers each corresponding to a test taken by a person from a particular definable classification and comprising an electronic representation of at least a portion of one or more test answer sheets;
• b) classification means for receiving a particular classification of each of the respondents or a group of the respondents and for electronically associating each of the particular classifications with the test answers for the corresponding respondent or group;
• and
• c) routing means for electronically selecting a test resolver for each of the answers based upon the particular classification of the corresponding respondent or group and for transmitting the answers to the selected test resolver.

A system for electronically and dynamically providing scoring rules to test resolvers to assist the test resolvers in scoring test items to which the scoring rules apply, the system comprising:

• a) receive means for electronically receiving a plurality of answers for test questions, the answers each comprising an electronic representation of at least a portion of a test answer sheet;
• b) storage means for electronically storing a plurality of scoring rules, each of the scoring rules relating to a procedure for scoring at least a particular one of the test questions;
• c) display means for selectively presenting the test answers to a test resolver;
• and
• d) rules means for selectively presenting to the test resolver the scoring rules related to the test question corresponding to the answer presented to the test resolver.