# US 2025/0174144 A1 · Automated Evaluation of Free-Form Answers and Generation of Actionable Feedback to Multidimensional Reasoning Questions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0174144 A1
- Document ID: US20250174144A1
- Status: Published application
- Kind: A1
- Application number: 18/754,026
- Filed: 2024-06-25
- Published: 2025-05-29
- Earliest claimed: 2022-08-15
- Assignees: BrainPOP IP LLC
- Applicants: BrainPOP IP LLC
- Inventors: Ilia P. Rushkin; Yigal Rosen; Sara Bakken; Michelle Newstadt
- Web page: https://patentgrep.com/patent/US20250174144A1

## CPC

- G09B 7/04: characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question or supplying a further explanation
- G06F 40/216: using statistical methods
- G06F 40/232: Orthographic correction, e.g. spell checking or vowelisation
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/30: Semantic analysis
- G06Q 50/20: Education
- G09B 5/14: with provision for individual teacher-student communication
- G09B 7/12: characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question or supplying further information
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

In an illustrative embodiment, methods and systems for automatically evaluating content of freeform text answers involve obtaining sections of text answering a multi-dimensional reasoning question, analyzing section content of each section using AI model(s), analyzing logical connections between the sections of text, and calculating score(s) corresponding to the freeform answer based on the analyses.

## Claims

### Claim 1

(canceled)

### Claim 2

A method for automatically evaluating freeform answers, the method comprising:
training an artificial intelligence (AI) model to evaluate freeform answers;
obtaining a freeform answer to a given multi-dimensional reasoning question, wherein the freeform answer is comprised of at least three sections, each section addressing a different dimension of the multi-dimensional answer;
converting each section of the freeform answer into a plurality of tokens, wherein each token represents a word or punctuation mark of the freeform answer,
formatting the plurality of tokens of each section of the freeform answer into a plurality of vectors, wherein the vectors are comprised of token sequences,
evaluating the plurality of vectors of each section of the freeform answer, wherein each section is evaluated individually according to its respective dimension of the multi-dimensional answer; and
calculating, based at least in part on the evaluation for each section, at least one score corresponding to the freeform answer.

### Claim 3 (depends on 2)

The method of claim 2, wherein the formatting further comprises:
creating a directed graph, wherein a plurality of vertices of the directed graph are comprised of the plurality of tokens and a plurality of edges of the directed graph are syntactic connections representing the grammatical relation between the tokens, and
formatting the directed graph into a plurality of vectors comprised of token sequences, wherein each vector of the plurality of vectors is an n-gram comprised of at least two vertices of the plurality of vertices and at least one edge of the plurality of edges connecting the at least two vertices.

### Claim 4 (depends on 2)

The method of claim 2, wherein the plurality of tokens are converted into an unordered list of tokens.

### Claim 5 (depends on 2)

The method of claim 2, wherein the plurality of tokens are formatted into a plurality of vectors comprised of words.

### Claim 6 (depends on 2)

The method of claim 2, wherein the freeform answer is organized according to a standardized framework.

### Claim 7 (depends on 6)

The method of claim 6, wherein the standardized framework is comprised of a claim section, an evidence section, and a reasoning section.

### Claim 8 (depends on 2)

The method of claim 2, further comprising evaluating the content of at least one section of the freeform answer.

### Claim 9 (depends on 2)

The method of claim 2, further comprising evaluating the grammar of at least one section of the freeform answer.

### Claim 10 (depends on 2)

The method of claim 2, further comprising evaluating the style of at least one section of the freeform answer.

### Claim 11 (depends on 2)

The method of claim 2, further comprising the step of evaluating the logical flow between at least two sections of the freeform answer.

### Claim 12 (depends on 2)

The method of claim 2, further comprising the step of continuously training the AI model with multi-dimensional freeform answers and their respective automatic evaluations.

### Claim 13 (depends on 2)

The method of claim 2, further comprising the step of providing the at least one score to a recommendation process for selecting a next learning activity for a learner.

### Claim 14 (depends on 2)

The method of claim 2, further comprising the step of providing the at least one score to a recommendation process for selecting the next multi-dimensional reasoning question.

### Claim 15

A method for formatting a freeform answer into a plurality of n-grams, the method comprising:
converting the freeform answer into a plurality of tokens, wherein each token represents a word or punctuation mark of the freeform answer, and
the plurality of tokens are organized into an unordered list;
creating a directed graph, wherein a plurality of vertices of the directed graph are comprised of the plurality of tokens and a plurality of edges of the directed graph are syntactic connections representing the grammatical relation between the tokens; and
formatting the directed graph into a plurality of vectors comprised of token sequences, wherein each vector of the plurality of vectors is an n-gram comprised of at least two vertices of the plurality of vertices and at least one edge of the plurality of edges connecting the at least two vertices.

### Claim 16 (depends on 15)

The method of claim 15, wherein each n-gram of the plurality of n-grams is comprised of two vertices and one connecting edge.

### Claim 17 (depends on 15)

The method of claim 15, wherein each n-gram of the plurality of n-grams is comprised of three vertices and one connecting edge between a first vertex and a second vertex, and a second connecting edge between the second vertex and a third vertex.

### Claim 18 (depends on 15)

The method of claim 15, wherein a given token of the plurality of tokens is represented in a plurality of vectors of the directed graph.

### Claim 19

A method for formatting a freeform answer into a plurality of n-grams, the method comprising:
converting the freeform answer into a plurality of tokens, wherein each token represents a word or punctuation mark of the freeform answer, and
creating a parsing tree, wherein a plurality of nodes of the parsing tree are comprised of the plurality of tokens and a plurality of branches of the parsing tree are syntactic connections representing the grammatical relation between the plurality of tokens,
formatting the parsing tree into a plurality of vectors comprised of token sequences, wherein each vector of the plurality of vectors is an n-gram comprised of at least one node and at least one metric derived from the structure of the parsing tree.

### Claim 20 (depends on 19)

The method of claim 19, wherein the at least one metric derived from the structure of the parsing tree is the number of nodes with no outgoing branches.

### Claim 21 (depends on 19)

The method of claim 19, wherein the at least one metric derived from the structure of the parsing tree is and average number of outgoing branches from a plurality of nodes.

### Claim 22 (depends on 19)

The method of claim 19, wherein the at least one metric derived from the structure of the parsing tree is the longest path between the at least one node and another at least one node.
