# US 2021/0343174 A1 · Unsupervised machine scoring of free-response answers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343174 A1
- Document ID: US20210343174A1
- Status: Published application
- Kind: A1
- Application number: 17/246,563
- Filed: 2021-04-30
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: Suffolk University
- Applicants: Suffolk University
- Inventors: David A. Colarusso
- Web page: https://patentgrep.com/patent/US20210343174A1

## CPC

- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 40/30: Semantic analysis
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Unsupervised machine scoring of free-response answers can be provided, eliminating the need to create a model answer. A scoring system can receive a set of free-response answers with associated response content and determine, from all the associated response content, a commonality content by identifying semantically related response content from the set of free-response answers. For each free-response answer, the scoring system can determine an amount of similarity between the associated response content for that free-response answer and the commonality content and assign a similarity value from the amount of similarity to that free-response answer. The amount of similarity indicates a degree of “correctness” of an answer and, according to an implementation, can be considered to be the distance of an embedding of a free-response answer from a vector-related average of all the free-response answers. The scoring system can provide the set of free-response answers and the corresponding assigned similarity values.

## Claims

### Claim 1

A method comprising:
receiving, at a computing system, a set of free-response answers, each free-response answer of the set of free-response answers comprising an associated response content;
determining, by the computing system, from the associated response content of the set of free-response answers, a commonality content by identifying semantically related response content from the set of free-response answers;
for each free-response answer of the set of free-response answers, determining by the computing system, an amount of similarity between the associated response content for that free-response answer and the commonality content;
assigning, by the computing system, a similarity value from the amount of similarity to that free-response answer; and
providing the set of free-response answers and their corresponding assigned similarity values.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining of the commonality content comprises:
generating a text embedding for the associated response content of each free-response answer;
determining a vector-related average of the generated text embeddings; and
determining a target answer according to the vector-related average.

### Claim 3 (depends on 2)

The method of claim 2, wherein the determining of the vector-related average of the generated text embeddings comprises determining a centroid or a medoid of the generated text embeddings.

### Claim 4 (depends on 2)

The method of claim 2, wherein the determining of the amount of similarity between the associated response content for that free-response answer and the commonality content comprises:
determining a distance between the text embedding for that free-response answer and the target answer.

### Claim 5 (depends on 4)

The method of claim 4, wherein the assigning of the similarity value from the amount of similarity to that free-response answer comprises:
assigning a value to each free-response answer based on the distance determined for that free-response answer.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining a ranked order of the set of free-response answers based on the assigned similarity values,
wherein the set of free-response answers having the corresponding assigned similarity values are provided in the ranked order.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
converting the assigned similarity values into Z-scores for each free-response answer in the set of free-response answers; and
translating the Z-scores to a grading scale.

### Claim 8

A system comprising:
a processing system;
a storage system; and
instructions stored on the storage system that when executed by the processing system direct the processing system to at least:
receive a set of free-response answers, each free-response answer of the set of free-response answers comprising an associated response content;
determine from the associated response content of the set of free-response answers, a commonality content by identifying semantically related response content from the set of free-response answers;
for each free-response answer of the set of free-response answers, determine an amount of similarity between the associated response content for that free-response answer and the commonality content;
assign a similarity value from the amount of similarity to that free-response answer; and
provide the set of free-response answers and their corresponding assigned similarity values.

### Claim 9 (depends on 8)

The system of claim 8, wherein the instructions to determine the commonality content direct the processing system to:
generate a text embedding for the associated response content of each free-response answer;
determine a vector-related average of the generated text embeddings; and
determine a target answer according to the vector-related average.

### Claim 10 (depends on 9)

The system of claim 9, wherein the instructions to determine the amount of similarity between the associated response content for that free-response answer and the commonality content direct the processing system to:
determine a distance between the text embedding for that free-response answer and the target answer.

### Claim 11 (depends on 10)

The system of claim 10, wherein the instructions to assign the similarity value from the amount of similarity to that free-response answer direct the processing system to:
assign a value to each free-response answer based on the distance determined for that free-response answer.

### Claim 12 (depends on 9)

The system of claim 9, wherein the target answer is a free-response answer from the set of free-response answers.

### Claim 13 (depends on 8)

The system of claim 8, wherein the instructions further direct the processing system to:
determine a ranked order of the set of free-response answers based on the assigned similarity values,
wherein the set of free-response answers having the corresponding assigned similarity values are provided in the ranked order.

### Claim 14

A computer-readable storage medium having instructions stored thereon that, when executed by a processing system, perform a method comprising:
receiving a set of free-response answers, each free-response answer of the set of free-response answers comprising an associated response content;
determining, from the associated response content of the set of free-response answers, a commonality content by identifying semantically related response content from the set of free-response answers;
for each free-response answer of the set of free-response answers, determining an amount of similarity between the associated response content for that free-response answer and the commonality content;
assigning a similarity value from the amount of similarity to that free-response answer; and
providing the set of free-response answers and their corresponding assigned similarity values.

### Claim 15 (depends on 14)

The medium of claim 14, wherein the determining of the commonality further comprises:
generating a text embedding for the associated response content of each free-response answer;
determining a vector-related average of the generated text embeddings; and
determining a target answer according to the vector-related average.

### Claim 16 (depends on 15)

The medium of claim 15, wherein the vector-related average is a centroid.

### Claim 17 (depends on 15)

The medium of claim 15, wherein the vector-related average is a medoid.

### Claim 18 (depends on 15)

The medium of claim 15, wherein the determining of the amount of similarity between the associated response content for that free-response answer and the commonality content comprises:
determining a distance between the text embedding for that free-response answer and the target answer.

### Claim 19 (depends on 18)

The medium of claim 18, wherein the assigning of the similarity value from the amount of similarity to that free-response answer comprises:
assigning a value to each free-response answer based on the distance determined for that free-response answer.

### Claim 20 (depends on 14)

The medium of claim 14, wherein the method further comprises:
converting the assigned similarity values into Z-scores for each free-response answer in the set of free-response answers; and
translating the Z-scores to a grading scale.
