Back to results
US 12,020,593 B2Granted
Automated evaluation of free-form answers and generation of actionable feedback to multidimensional reasoning questions
AssigneeBrainPOP IP LLC
InventorsIlia P. Rushkin, Yigal Rosen, Sara Bakken, Michelle Newstadt
Life of this patent
Estimating the term…
Abstract
In an illustrative embodiment, methods and systems for automatically evaluating content of freeform text answers involve obtaining sections of text answering a multi-dimensional reasoning question, analyzing section content of each section using AI model(s), analyzing logical connections between the sections of text, and calculating score(s) corresponding to the freeform answer based on the analyses.
Claims19
A system for automatically evaluating content of freeform answers, the system comprising:
- a non-volatile data store comprising a plurality of multi-dimensional reasoning questions related to one or more subjects;
- a plurality of artificial intelligence (AI) models, each respective AI model of the plurality of AI models being fine-tuned in evaluating freeform answers according to a standardized framework for organizing freeform answers by providing a respective set of historic answers and a respective set of corresponding answer scores to the respective AI model;
- and
- processing circuitry configured to perform operations comprising
- obtaining a freeform answer to a given multi-dimensional reasoning question of the plurality of multi-dimensional reasoning questions, the freeform answer comprising a corresponding section text for each respective section of a predetermined section structure of the freeform answer, wherein
- the corresponding section text of each respective section comprises one or more sentences, and
- the predetermined section structure comprises at least three sections defined by the standardized framework, each section of the at least three sections having a different goal of a set of goals of the predetermined section structure,
- for each respective section of the predetermined section structure,
- performing a respective section content evaluation by communicating at least one engineered model prompt to each model of one or more AI models of the plurality of AI models to instruct evaluation of the corresponding section text for that respective section in view of a given goal of the set of goals for the respective section, thereby obtaining at least one section content score,
- identifying another AI model of the plurality of AI models configured to evaluate logical connections between the section text of adjacent sections of the predetermined section structure,
- performing a logical connection evaluation by communicating one or more engineered model prompts to the another AI model to instruct evaluation of logical flow between the corresponding section texts of respective sections that are adjacent in the predetermined section structure, thereby obtaining at least one logical connection score, and
- calculating, using the at least one section content score and the at least one logical connection score, at least one overall score corresponding to the freeform answer.
9 more elements
A method for automatically evaluating content of freeform answers, the method comprising:
- tuning, by processing circuitry, a plurality of artificial intelligence (AI) models to evaluate freeform answers according to a standardized framework for organizing freeform answers, wherein the tuning of each respective AI model of the plurality of AI models comprises
- providing a respective set of historic answers and a respective set of corresponding answer scores to the respective AI model;
- obtaining, via a text entry user interface of an electronic learning platform, corresponding section text for each respective section of a plurality of text sections of a freeform answer to a given multi-dimensional reasoning question of a plurality of multi-dimensional reasoning questions of the electronic learning platform, wherein
- the freeform answer is organized according to the standardized framework and comprises a predetermined section structure, wherein
- the predetermined section structure comprises at least three sections, each section of the at least three sections having a different goal of a set of goals of the predetermined section structure, and
- the corresponding section text of each respective section comprises one or more sentences;
- formatting, by the processing circuitry, the corresponding section text for each respective text section of the plurality of text sections for submission to at least one respective AI model of the plurality of AI models;
- for each respective text section of the plurality of text sections, submitting, by the processing circuitry, the formatted corresponding section text and at least one engineered model prompt for the respective text section to a first at least one AI model of the plurality of AI models to form a content evaluation of the respective text section in view of a given goal of the set of goals for the respective section;
- submitting, by the processing circuitry, the formatted corresponding section texts of each respective text section of the plurality of text sections and one or more engineered model prompts to a second at least one AI model of the plurality of AI models to instruct evaluation of logical flow between the formatted corresponding section texts of adjacent text sections in the plurality of text sections;
- and
- calculating, based at least in part on the respective content evaluation for each of the respective text sections of the plurality of text sections and the evaluation of the logical flow, at least one score corresponding to the freeform answer.
8 more elements
Dependencies are inferred from the claim text.