# US 11,443,140 B2 · Systems and methods for automated machine learning model training for a custom authored prompt

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,140 B2
- Document ID: US11443140B2
- Status: Granted
- Kind: B2
- Application number: 16/280,984
- Filed: 2019-02-20
- Published: 2019-08-22
- Granted: 2022-09-13
- Earliest claimed: 2018-02-20
- Assignees: Pearson Education, Inc.
- Applicants: Pearson Education, Inc.
- Inventors: Scott Hellman; Lee Becker; Samuel Downs; Alok Baikadi; William Murray; Kyle Habermehl; Peter Foltz; Mark Rosenstein
- Web page: https://patentgrep.com/patent/US11443140B2

## CPC

- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 9/451: Execution arrangements for user interfaces
- G06F 16/40: of multimedia data, e.g. slideshows comprising image and additional audio data (retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70)
- G06F 16/904: Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957)
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 40/205: Parsing
- G06F 40/30: Semantic analysis
- G06N 20/00: Machine learning
- G09B 7/06: of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
- G06F 8/38: for implementing user interfaces
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

## Claims

### Claim 1

A system for training a model for a custom authored prompt, the system comprising:
a memory comprising:
a content library database comprising a plurality of prompts, and
a model database comprising at least one prompt evaluation model trained to evaluate prompts; and
at least one processor configured to:
receive a prompt;
parse the prompt to identify a plurality of prompt evaluation portions;
identify pre-existing data relevant to at least one prompt evaluation portion of the prompt evaluation portions;
train a response evaluation model for evaluating responses to the prompt at least in part based on the pre-existing data;
provide the prompt to a user;
receive a response to the provided prompt;
generate an evaluation of the received response;
determine sufficiency of the training of the response evaluation model;
update the training based on the received response and on the evaluation of the received response to determine new training data when the training of the response evaluation model is insufficient;
determine a plurality of responses and a plurality of evaluations of the plurality of received responses;
determine an ordering to the received responses;
train based on the ordering of the received responses; and
generate a training indicator, wherein the training indicator is configured to provide a graphical depiction of a degree to which the response evaluation model is trained based upon the updating of the training of the response evaluation model.

### Claim 2 (depends on 1)

The system of claim 1, wherein parsing the prompt comprises identifying a complexity of each of the plurality of prompt evaluation portions associated with the received prompt.

### Claim 3 (depends on 2)

The system of claim 2,
wherein the pre-existing data comprises at least one of:
a pre-existing model trained to evaluate responses to another prompt, and
pre-existing response data generated from responses to other prompts.

### Claim 4 (depends on 3)

The system of claim 3, wherein the at least one processor is further configured to identify a creator of the received prompt, and wherein the at least one of:
the pre-existing model, and the pre-existing response data, are identified based on the creator of the received prompt.

### Claim 5 (depends on 4)

The system of claim 4, wherein identifying pre-existing response data comprises identifying response data corresponding to at least one of the plurality of prompt evaluation portions via a similarity score.

### Claim 6 (depends on 1)

The system of claim 1, wherein the at least one processor is further configured to control the training indicator to reflect the degree to which the response evaluation model is trained subsequent to the updating of the training of the response evaluation model.

### Claim 7

A method of training a model for a custom authored prompt, the method comprising:
receiving a prompt;
parsing the prompt to identify a plurality of prompt evaluation portions associated with the received prompt;
identifying pre-existing data relevant to at least one of the prompt evaluation portions;
training a response evaluation model for evaluating responses to the prompt at least in part based on the pre-existing data;
receiving a response to the prompt;
generating an evaluation of the received response;
determine sufficiency of the training of the response evaluation model;
update the training based on the received response and on the evaluation of the received response to determine new training data when the training of the response evaluation model is insufficient;
determining a plurality of responses and a plurality of evaluations of the plurality of received responses;
training based on an ordering of the received responses; and
generating a training indicator, wherein the training indicator is configured to provide a graphical depiction of a degree to which the response evaluation model is trained based upon the updating of the training of the response evaluation model.

### Claim 8 (depends on 7)

The method of claim 7, wherein parsing the prompt comprises identifying a complexity of each of the plurality of prompt evaluation portions associated with the received prompt.

### Claim 9 (depends on 7)

The method of claim 7, wherein the pre-existing data comprises at least one of: a pre-existing model trained to evaluate responses to another prompt, and pre-existing response data generated from responses to other prompts.

### Claim 10 (depends on 9)

The method of claim 9, further comprising identifying a creator of the received prompt, and wherein the at least one of: the pre-existing model, and the pre-existing response data, are identified based on the creator of the received prompt.

### Claim 11 (depends on 9)

The method of claim 9, wherein identifying pre-existing response data comprises identifying response data corresponding to at least one of the plurality of prompt evaluation portions via a similarity score.

### Claim 12 (depends on 7)

The method of claim 7, further comprising:
providing the prompt to a user.

### Claim 13 (depends on 7)

The method of claim 7, wherein updating the training comprises determining the ordering to the received responses.

### Claim 14 (depends on 13)

The method of claim 13, further comprising controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.