A computer-based method for improving computer-generated exams and generating exam question sets suited to separating test candidates of different skill levels, the method comprising the steps of:
- storing a number of questions in at least one predetermined topic in a database, each question being associated with a data set comprising prior use information related to answers given to the question in previous tests presented to test candidates, said database being stored on at least one data storage device and updated to reflect empirical data collected from previous exams to find selectivity, facility and reliability of question and question sets presented to test candidates, wherein the empirical data for each question on the computer generated exams is collected to find the selectivity, reliability and facility of the questions and the database is updated to reflect the collected empirical data, said database further comprising unused questions including newly added questions;
- randomly selecting by means executing on a computer questions within one or more of the predetermined topics from said database;
- evaluating by means executing on a computer, for possible inclusion in a new question set for presentation to test candidates, the selected questions relative to predetermined requirements applied to such prior use information on the selected questions and their corresponding answer alternatives in said data set;
- and
- discarding from possible inclusion in the new question set selected questions not fulfilling said predetermined requirements, thereby generating from the database of questions the new question set for presentation to test candidates, wherein selected questions that do not fit a chosen profile regarding selectivity and facility are discarded from the new question set, such that a controlled distribution of difficulty is obtained;
- wherein the evaluating means is adapted for discarding the selected questions not fulfilling the predetermined requirements by fitting a number of correct answers per failure in the selected questions with a predetermined distribution, so as to ensure the controlled distribution of difficulty in the new question set;
- and
- wherein the evaluating means is further adapted to find a coefficient indicating a quality of the questions in the database, the database including the questions associated with prior use information and the unused questions including newly added questions, by calculating an actual reliability of the new question set after the new question set has been in use and comparing the actual reliability of the new question set after it has been in use to a calculated predicted reliability to find the coefficient;
- based on:
- constructing a set of probable question set answers, and, with the set of probable question set answers, calculating a measure for predicted reliability of the new question set, assuming the new question set to have a same distribution of characteristics as a known set of the questions having known reliability;
- and
- correcting the predicted reliability of the new question set, relative to the actual reliability of the new question set after the new question set has been in use, wherein the empirical data are used to discriminate the questions and produce an exam question set with higher quality than a randomly selected set.