Mathematical language processing: automatic grading and feedback for open response mathematical questions

A non-transitory memory medium for operating a computer in order to automatically grade solutions submitted by learners in response to a question, wherein the memory medium stores program instructions, wherein the program instructions, when executed by the computer, cause the computer to implement:

• receiving the solutions via a computer network, wherein each of the received solutions is provided by a respective one of the learners, and includes one or more mathematical expressions, wherein a majority of the received solutions include more than one mathematical expression;
• assembling a list of unique expressions occurring in a union of the received solutions;
• for each of the received solutions, automatically determining a corresponding feature vector based on the mathematical expressions occurring in the received solution, wherein the feature vector for each received solution is a binary valued vector whose components correspond respectively to expressions in the list of unique expressions;
• automatically determining measures of similarity between the received solutions, wherein said automatically determining measures of similarity between the received solutions includes, for each pair of the received solutions, automatically determining a measure of similarity between the received solutions of the pair based on the corresponding pair of feature vectors;
• automatically clustering the received solutions into K>1 clusters based on the similarity measures;
• for each of the K clusters:
• receiving grading input from one or more graders via the computer network, wherein the grading input indicates a grade to be assigned to a representative received solution in the cluster;
• and
• automatically assigning the grade of the representative received solution in the cluster to the other received solutions in the cluster;
• and
• storing the grades for the received solutions in a memory.

A method for operating a computer in order to automatically grade solutions submitted by learners in response to a question, the method comprising:

• receiving the solutions via a computer network, wherein each of the received solutions is provided by a respective one of the learners, and includes one or more mathematical expressions, wherein a majority of the received solutions include more than one mathematical expression;
• assembling a list of unique expressions occurring in a union of the received solutions;
• for each of the received solutions, automatically determining a corresponding feature vector based on the mathematical expressions occurring in the received solution, wherein the feature vector for each received solution is a binary valued vector whose components correspond respectively to expressions in the list of unique expressions;
• automatically performing a probabilistic clustering algorithm based on the feature vectors of the received solutions to determine:
• (a) {circumflex over (K)} conditional probability distributions corresponding to {circumflex over (K)} respective clusters, wherein each of the {circumflex over (K)} conditional probability distributions characterizes a corresponding distribution of generic feature vector y given membership in the corresponding cluster;
• and
• (b) for each of the received solutions, an assignment of the received solution to a corresponding one of the {circumflex over (K)} clusters;
• for each of the {circumflex over (K)} clusters, receiving grading input from one or more graders via the computer network, wherein each grading input indicates a grade to be assigned to a representative received solution in a respective cluster of the {circumflex over (K)} clusters;
• for each of the received solutions other than the representative received solutions:
• automatically assigning a grade to the received solution based on the grades assigned to the representative received solutions and the {circumflex over (K)} conditional probability distributions evaluated on the feature vector of the received solution;
• and
• storing the grades for the received solutions in a memory.