Automated Evaluation of Free-Form Answers and Generation of Actionable Feedback to Multidimensional Reasoning Questions
AssigneeBrainPOP IP LLC
InventorsIlia P. Rushkin, Yigal Rosen, Sara Bakken, Michelle Newstadt
Life of this patent
Estimating the term…
Abstract
In an illustrative embodiment, methods and systems for automatically evaluating content of freeform text answers involve obtaining sections of text answering a multi-dimensional reasoning question, analyzing section content of each section using AI model(s), analyzing logical connections between the sections of text, and calculating score(s) corresponding to the freeform answer based on the analyses.
Claims22
(canceled)
A method for automatically evaluating freeform answers, the method comprising:
- training an artificial intelligence (AI) model to evaluate freeform answers;
- obtaining a freeform answer to a given multi-dimensional reasoning question, wherein the freeform answer is comprised of at least three sections, each section addressing a different dimension of the multi-dimensional answer;
- converting each section of the freeform answer into a plurality of tokens, wherein each token represents a word or punctuation mark of the freeform answer,
- formatting the plurality of tokens of each section of the freeform answer into a plurality of vectors, wherein the vectors are comprised of token sequences,
- evaluating the plurality of vectors of each section of the freeform answer, wherein each section is evaluated individually according to its respective dimension of the multi-dimensional answer;
- and
- calculating, based at least in part on the evaluation for each section, at least one score corresponding to the freeform answer.
A method for formatting a freeform answer into a plurality of n-grams, the method comprising:
- converting the freeform answer into a plurality of tokens, wherein each token represents a word or punctuation mark of the freeform answer, and
- the plurality of tokens are organized into an unordered list;
- creating a directed graph, wherein a plurality of vertices of the directed graph are comprised of the plurality of tokens and a plurality of edges of the directed graph are syntactic connections representing the grammatical relation between the tokens;
- and
- formatting the directed graph into a plurality of vectors comprised of token sequences, wherein each vector of the plurality of vectors is an n-gram comprised of at least two vertices of the plurality of vertices and at least one edge of the plurality of edges connecting the at least two vertices.
A method for formatting a freeform answer into a plurality of n-grams, the method comprising:
- converting the freeform answer into a plurality of tokens, wherein each token represents a word or punctuation mark of the freeform answer, and
- creating a parsing tree, wherein a plurality of nodes of the parsing tree are comprised of the plurality of tokens and a plurality of branches of the parsing tree are syntactic connections representing the grammatical relation between the plurality of tokens,
- formatting the parsing tree into a plurality of vectors comprised of token sequences, wherein each vector of the plurality of vectors is an n-gram comprised of at least one node and at least one metric derived from the structure of the parsing tree.
Dependencies are inferred from the claim text.