# US 11,443,647 B2 · Systems and methods for assessment item credit assignment based on predictive modelling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,647 B2
- Document ID: US11443647B2
- Status: Granted
- Kind: B2
- Application number: 16/786,044
- Filed: 2020-02-10
- Published: 2020-08-13
- Granted: 2022-09-13
- Earliest claimed: 2019-02-08
- Assignees: Pearson Education, Inc.
- Applicants: Pearson Education, Inc.
- Inventors: Mark E. Liedtke; Sumona J. Routh; Clayton Tong; Daniel L. Ensign; Victoria Kortan; Srirama Kolla
- Web page: https://patentgrep.com/patent/US11443647B2

## CPC

- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G06F 16/156: {Query results presentation}
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G09B 7/00: Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F)
- G09B 7/06: of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
- G06N 3/08: Learning methods
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are provided by which an adaptive learning engine may be executed to determine the probability that a given user will respond correctly to a given assessment item of a digital assessment on their first attempt. The adaptive learning engine may apply one or more machine learning models to feature data corresponding to the user and the assessment item in order to determine the probability. The feature data may be calculated periodically and/or in real time or near-real time according to a machine learning model definition based on assessment data corresponding to the user's activity and/or based on responses submitted globally by users to the assessment item and/or to content related to the assessment item. Based on the correct first attempt probability, the adaptive learning engine may identify and recommend assessment items for which a user should be preemptively assigned credit.

## Claims

### Claim 1

A system comprising:
a computer processor;
an electronic communication network, the computer processor being in electronic communication with an external server via the electronic communication network;
a feature data store in electronic communication with the computer processor;
a model definition data store in electronic communication with the computer processor; and
a computer memory in electronic communication with the computer processor and configured to store computer-readable instructions which, when executed, cause the computer processor to:
receive, with an adaptive learning engine service, a request from the external server, the request including a user identifier corresponding to a user, an assessment item identifier corresponding to an assessment item of a digital assessment, and request data;
retrieve, with a feature retrieval service, a machine learning model definition from the model definition data store based on the request data;
retrieve, with the feature retrieval service, first feature data from the feature data store based on the user identifier, the assessment item identifier, and the machine learning model definition;
execute, with a machine learning model service, a machine learning model corresponding to the machine learning model definition to process the first feature data to determine a correct first attempt probability for the user for the assessment item;
determine, with a prediction generator, that the correct first attempt probability exceeds a predetermined threshold; and
send, with the adaptive learning engine service, a recommendation to the external server indicating that credit for the assessment item should be automatically given to the user without requiring the user to submit a response to the assessment item.

### Claim 2 (depends on 1)

The system of claim 1, further comprising:
an attempt data store in electronic communication with the computer processor; and
a user activity analytics service configured to monitor, in near-real-time, interactions between user devices and the system and to cause assessment data representing the interactions to be stored in the attempt data store.

### Claim 3 (depends on 2)

The system of claim 2, wherein the computer-readable instructions, when executed, cause the computer processor to:
retrieve, with the feature retrieval service, the assessment data from the attempt data store;
calculate, with the feature retrieval service, second feature data from the assessment data based on the machine learning model definition; and
calculate, with the feature retrieval service, the second feature data based on the assessment data, wherein the machine learning model service processes both the second feature data and the first feature data with the machine learning model to determine the correct first attempt probability.

### Claim 4 (depends on 3)

The system of claim 3, wherein the assessment data retrieved by the feature retrieval service represents one or more responses to assessment items submitted by the user and one or more grades corresponding to the one or more responses.

### Claim 5 (depends on 4)

The system of claim 4, further comprising:
a global data store configured to store global assessment data representing a plurality of users and at least one course, wherein a course of the at least one course is associated with the assessment item; and
a global feature analytics service configured to:
retrieve the global assessment data from the global data store;
generate the first feature data based on the global assessment data; and
cause the first feature data to be stored in the feature data store.

### Claim 6 (depends on 5)

The system of claim 5, wherein the first feature data includes global correct first attempt rates calculated based on the global assessment data, and wherein the second feature data includes user correct first attempt rates calculated based on the assessment data.

### Claim 7 (depends on 1)

The system of claim 1, wherein the recommendation is sent to the external server upon determining that the correct first attempt probability exceeds the predetermined threshold.

### Claim 8

A system comprising:
at least one processor; and
at least one memory device configured to store computer-readable instructions which, when executed, cause the at least one processor to:
receive a request from an external server;
analyze the request to identify a user and a plurality of assessment items corresponding to a digital assessment;
retrieve a machine learning model definition based on the request;
retrieve first feature data based on the user, the assessment item, and the machine learning model definition;
execute a machine learning model corresponding to the machine learning model definition to process at least the first feature data to determine a plurality of correct first attempt probabilities for the user for the plurality of assessment items, wherein each of the plurality of correct first attempt probabilities represents a respective probability that a corresponding response submitted by the user to a corresponding assessment item of the plurality of assessment items will be correct;
identify a subset of the plurality of correct first attempt probabilities having values exceeding a predetermined threshold; and
cause credit to automatically be recorded for the user for a subset of the plurality of assessment items corresponding to the subset of the plurality of correct first attempt probabilities.

### Claim 9 (depends on 8)

The system of claim 8, wherein the computer-readable instructions, when executed, cause the at least one processor to:
retrieve assessment data representing, in near-real-time, interactions between the user and a content distribution network; and
calculate second feature data based on the assessment data according to instructions included in the machine learning model definition.

### Claim 10 (depends on 9)

The system of claim 9, wherein, to execute the machine learning model, the computer-readable instructions, when executed, further cause the at least one processor to:
execute the machine learning model to process the first feature data and the second feature data to determine the correct first attempt probability for the user for the assessment item.

### Claim 11 (depends on 10)

The system of claim 10, wherein the computer-readable instructions, when executed, cause the at least one processor to:
retrieve global assessment data representing a plurality of users and at least one course, wherein a course of the at least one course is associated with the digital assessment;
generate the first feature data based on the global assessment data; and
cause the first feature data to be stored in a feature data store of the at least one memory device.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first feature data includes a plurality of global correct first attempt rates for the plurality of users calculated based on the global assessment data, and wherein the second feature data includes user correct first attempt rates for the user calculated based on the assessment data.

### Claim 13 (depends on 8)

The system of claim 8, wherein the credit is automatically recorded without the subset of the plurality of assessment items being presented at a user device associated with the user.

### Claim 14

A method comprising:
receiving, with at least one computer processor, a request from an external server;
analyzing, with the at least one computer processor, the request to identify a user and a plurality of assessment items corresponding to a digital assessment;
retrieving, with the at least one computer processor, a machine learning model definition based on the request;
retrieving, with the at least one computer processor, first feature data based on the user, the assessment item, and the machine learning model definition;
executing, with the at least one computer processor, a machine learning model corresponding to the machine learning model definition to process at least the first feature data to determine a plurality of correct first attempt probabilities for the user for the plurality of assessment items, wherein each of the plurality of correct first attempt probabilities represents a respective probability that a corresponding response submitted by the user to a corresponding assessment item of the plurality of assessment items will be correct;
identifying, with the at least one computer processor, a subset of the plurality of correct first attempt probabilities having values exceeding a predetermined threshold; and
sending, with the at least one computer processor to the external server, a recommendation to automatically assign credit for the user for a subset of the plurality of assessment items corresponding to the subset of the plurality of correct first attempt probabilities.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
retrieving, with the at least one computer processor, assessment data representing, in near-real-time, interactions between the user and a content distribution network; and
calculating, with the at least one computer processor, second feature data based on the assessment data according to instructions included in the machine learning model definition.

### Claim 16 (depends on 15)

The method of claim 15, wherein executing the machine learning model further comprises:
executing, with the at least one computer processor, the machine learning model to process the first feature data and the second feature data to determine the correct first attempt probability for the user for the assessment item.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
retrieving, with the at least one computer processor, global assessment data representing a plurality of users and at least one course, wherein a course of the at least one course is associated with the digital assessment;
generating, with the at least one computer processor, the first feature data based on the global assessment data; and
causing, with the at least one computer processor, the first feature data to be stored in a feature data store of a memory device that is in electronic communication with the at least one computer processor.

### Claim 18 (depends on 17)

The method of claim 17, wherein the first feature data includes a plurality of global correct first attempt rates for the plurality of users calculated based on the global assessment data, and wherein the second feature data includes user correct first attempt rates for the user calculated based on the assessment data.

### Claim 19 (depends on 14)

The method of claim 14, wherein the credit is automatically recorded without the subset of the plurality of assessment items being presented at a user device associated with the user.

### Claim 20 (depends on 14)

The method of claim 14, wherein the recommendation is sent to the external server upon identifying the subset of the plurality of correct first attempt probabilities having values exceeding the predetermined threshold.