# US 2022/0036489 A1 · Recommendation engine for testing conditions based on evaluation of test entity scores

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036489 A1
- Document ID: US20220036489A1
- Status: Published application
- Kind: A1
- Application number: 16/941,378
- Filed: 2020-07-28
- Published: 2022-02-03
- Earliest claimed: 2020-07-28
- Assignees: NCS Pearson, Inc.
- Applicants: NCS Pearson, Inc.
- Inventors: Joseph BRUTSCHE; Darrick JENSEN; Michael NEALIS; Peter PASCALE; Vladan PULEC
- Web page: https://patentgrep.com/patent/US20220036489A1

## CPC

- G06Q 50/205: {Education administration or guidance}
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/0635: Risk analysis of enterprise or organisation activities
- G06Q 10/10: Office automation; Time management
- G06Q 50/20: Education
- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G06N 3/08: Learning methods
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Systems and methods may involve processing of entity data by machine learning models to produce one or more entity and/aggregate risk scores and/or aggregate anticipated risk scores, which may be compared to one or more thresholds to determine when one or more predefined actions should be taken. The entity data may be collected for various entities related to an exam registration and delivery process, which may include a candidate, an exam, a test center, an exam registration event, a proctor, and an exam delivery event. The exam registration and delivery process may include multiple states—each being associated with a different set of entities. Aggregate risk scores for a given state may be calculated using only entity data for the set of entities associated with that state. The predetermined actions taken may also be dependent on the current state.

## Claims

### Claim 1

A system comprising:
a data store coupled to a computer network and storing:
a plurality of entity data associated with an anticipated testing activity comprising:
an exam data;
a test center data associated with a test center at which the anticipated testing activity will occur;
a candidate data associated with:
each of a plurality of candidates anticipated to be involved in the anticipated testing activity; and
a historical testing data associated with each of the plurality of candidates;
a country data; and
a proctor data;
a machine learning model;
a plurality of recommended predefined actions;
a threshold, set by an administrator of the exam data and representing a desired level of trustworthiness which, if exceeded by an anticipated situational risk score, triggers a recommendation for at least one predefined action in the plurality of predefined actions; and
a server comprising a server hardware computing device coupled to a network and including at least one processor executing instructions within a memory that, when executed, cause the system to:
receive, from the administrator, the threshold;
select, from the data store, the machine learning model, and at least one entity data, in the plurality of entity data, associated with the anticipated testing activity;
perform a risk assessment of the at least one entity data, indicating a risk level associated with the anticipated testing activity and comprising:
inputting the at least one entity data as a machine learning parameter into the machine learning model;
receiving, as output from the machine learning model, the anticipated situational risk score, indicating a probability of an anticipated situational risk during the anticipated testing activity, and associated with, and based on at least one metric derived from, the at least one entity data; and
comparing the anticipated situational risk score to the threshold;
responsive to the anticipated situational risk score exceeding the threshold, update a database entry associated in the data store with the at least one entity data to include the at least one recommended predefined action in the plurality of recommended predefined actions, the at least one recommended predefined action providing a level of trustworthiness at least as high as the threshold.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a distributed computing system including a plurality of data repositories connected to the data store through the computer network, wherein:
the plurality of entity data is distributed among the data store and the plurality of data repositories; and
the plurality of data repositories are operated by:
an administrative entity operating the system;
a testing facility; or
a trusted third party.

### Claim 3 (depends on 1)

The system of claim 1, wherein the machine learning model is trained via a linear regression analysis or a classification training.

### Claim 4 (depends on 3)

The system of claim 3, further comprising a neural network used by the machine learning model, wherein the neural network includes a plurality of nodes configured to perform the linear regression analysis.

### Claim 5 (depends on 4)

The system of claim 4, further comprising a plurality of known-good testing activity data and a plurality of known-bad testing activity data stored in the data store, wherein the neural network performs the linear regression analysis using a plurality of samples from the plurality of known-good activity data.

### Claim 6

A method comprising:
receiving, by a server comprising a server computing device coupled to a computer network and including a processor executing instructions within a memory, a threshold set by an administrator of an exam data and representing a desired level of trustworthiness set by the administrator for an anticipated testing activity which, if exceeded by an anticipated situational risk score, triggers a recommendation for at least one recommended predefined action in a plurality of recommended predefined actions stored in a data store coupled to the computer network;
selecting, by the server from the data store, at least one entity data associated with the anticipated testing activity, the at least one entity data being selected from a plurality of entity data stored in the data store and comprising:
the exam data;
a test center data associated with a test center at which the anticipated testing activity will occur;
a candidate data associated with:
each of a plurality of candidates anticipated to be involved in the anticipated testing activity; and
a historical testing data associated with each of the plurality of candidates;
a country data; and
a proctor data;
performing, by the server, a risk assessment of each of the at least one entity data, indicating a risk level associated with the anticipated testing activity, and comprising:
selecting, from the data store, a machine learning model stored in the data store;
inputting the at least one entity data as a machine learning parameter into the machine learning model;
receiving, as output from the machine learning model, the anticipated situational risk score, indicating a probability of an anticipated situational risk during the anticipated testing activity, and associated with, and based on, at least one metric derived from the at least one entity data; and
comparing the anticipated situational risk score to the threshold;
responsive to the anticipated situational risk score exceeding the threshold, updating, by the server, a database entry, associated in the data store with the at least one entity data, to include the at least one recommended predefined action in the plurality of recommended predefined actions, the at least one recommended predefined action providing a level of trustworthiness at least as high as the threshold.

### Claim 7 (depends on 6)

The method of claim 6, wherein the exam data comprises:
an exam type for an exam associated with the anticipated testing activity;
a stakes rating for the exam;
an item bank size for an item bank associated with the exam;
a refresh rate for the item bank;
an item bank age for the item bank;
a delivery mechanism for the exam; and
at least one web monitoring result, indicating whether an exam content for the exam has been published online.

### Claim 8 (depends on 6)

The method of claim 6, wherein the test center data is selected from the group consisting of:
a name of the test center associated with, and anticipated to be used during, the anticipated testing activity;
a facility type of the test center;
at least one anticipated identification requirement at the test center;
a country or other geographical location associated with the test center, or in which the test center resides;
an age of the test center;
an IP address associated with the test center;
a geolocation of the test center, based on the IP address;
an exam registration history for the test center;
a software update compliance indicator for the test center;
an exam result pattern indicator for the test center;
a registration pattern indicator for the test center;
a geolocation mismatch indicator for the test center; and
a number of test centers located within a predefined radius from the test center.

### Claim 9 (depends on 6)

The method of claim 6, wherein the candidate data is selected from the group consisting of:
a name of a candidate associated with, and anticipated to be involved with, the testing activity;
an email address of the candidate;
a country of residence of the candidate;
a third-party demographic data associated with the candidate;
an Internet Protocol (IP) address associated with a user device operated by the candidate, the IP address being determined by a geolocation of the user device;
a bot behavior indicator associated with the IP address;
a historical fraudulent behavior indicator associated with the candidate;
an account takeover risk indicator associated with the candidate, the account takeover risk indicator being based on an account activity of a user profile associated with the candidate;
an age of the account; and
an exam registration history associated with the candidate.

### Claim 10 (depends on 9)

The method of claim 9, wherein the candidate data further comprises an anticipated class of candidates, according to:
an anticipated age range of at least one candidate anticipated to be involved with the testing activity;
an anticipated country of origin for the at least one candidate; and
at least one anticipated background of the at least one candidate, including:
at least one prior exam taken by the at least one candidate; and
a pass/fail rate of the at least one candidate.

### Claim 11 (depends on 6)

The method of claim 6, wherein the country data is selected from the group consisting of:
a candidate country in which a candidate associated with the anticipated testing activity is located;
an anticipated testing activity country in which the anticipated testing activity is located; and
a plurality of zones associated with the anticipated testing activity, and further associated with at least one legal restriction for the testing.

### Claim 12 (depends on 6)

The method of claim 6, wherein the proctor data is selected from the group consisting of:
an employment type indicator indicating whether a proctor associated with the entity data is an employee of a test center associated with the anticipated testing activity, or is a contractor;
a history of exams supervised by the proctor;
a total number of years that the proctor has been supervising exam delivery;
an employment history of the proctor;
a proctor fraud history indicating whether the proctor is associated with at least one previous incident of fraud;
a certification status of the proctor; and
at least one certification exam result for the proctor.

### Claim 13

A system comprising a server, comprising a server computing device coupled to a computer network and including at least one processor executing instructions within a memory which, when executed, cause the system to:
receive a threshold set by an administrator of an exam data and representing a desired level of trustworthiness set by the administrator for an anticipated testing activity which, if exceeded by an anticipated situational risk score, triggers a recommendation for at least one recommended predefined action in a plurality of recommended predefined actions stored in a data store coupled to the computer network;
select, from the data store, at least one entity data associated with the anticipated testing activity, the at least one entity data being selected from a plurality of entity data stored in the data store and comprising:
the exam data;
a test center data associated with a test center at which the anticipated testing activity will occur;
a candidate data associated with:
each of a plurality of candidates anticipated to be involved in the anticipated testing activity; and
a historical testing data associated with each of the plurality of candidates;
a country data; and
a proctor data;
perform a risk assessment of each of the at least one entity data, indicating a risk level associated with the anticipated testing activity, and comprising:
selecting, from the data store, a machine learning model stored in the data store;
inputting the at least one entity data as a machine learning parameter into the machine learning model;
receiving, as output from the machine learning model, the anticipated situational risk score, indicating a probability of an anticipated situational risk during the anticipated testing activity associated with, and based on at least one metric derived from, the at least one entity data; and
comparing the anticipated situational risk score to the threshold;
responsive to the anticipated situational risk score exceeding the threshold, updating, by the server, a database entry, associated in the data store with the at least one entity data, to include the at least one recommended predefined action in the plurality of predefined actions, the at least one recommended predefined action providing a level of trustworthiness at least as high as the threshold.

### Claim 14 (depends on 13)

The system of claim 13, wherein the server is further configured, responsive to the anticipated situational risk score exceeding the threshold, to automatically update the database entry to include a recommendation for:
a specific class of testing center for the anticipated testing activity; or
a reduction to a number of testing centers, according to a geographic region or the specific class of testing centers, for the anticipated testing activity.

### Claim 15 (depends on 13)

The system of claim 13, wherein the server is further configured, responsive to the anticipated situational risk score exceeding the threshold, to automatically update the database entry to include a recommendation for at least one identification check protocol.

### Claim 16 (depends on 15)

The system of claim 15, wherein the at least one identification check protocol comprises:
a first requirement for a photo identification for a candidate associated with the anticipated testing activity;
a second requirement for at least two forms of identification for the candidate; or
an ongoing identification validation during the anticipated testing activity.

### Claim 17 (depends on 13)

The system of claim 13, wherein the server is further configured, responsive to the anticipated situational risk score exceeding the threshold, to automatically update the database entry to include at least one recommendation for at least one proctor solution comprising:
an increased ratio of human proctors to a plurality of candidates associated with the anticipated testing activity; or
a use or non-use of an automated proctoring solution.

### Claim 18 (depends on 13)

The system of claim 13, wherein the server is further configured, responsive to the anticipated situational risk score exceeding the threshold, to automatically update the database entry to include a recommendation for storing captured data comprising a video or audio data during the anticipated testing activity.

### Claim 19 (depends on 13)

The system of claim 13, wherein the server is further configured, responsive to the anticipated situational risk score exceeding the threshold, to automatically update the database entry to include a recommendation for an increased security for an administration of the anticipated testing activity, the increased security addressing:
a high risk of an intellectual property associated with the exam being stolen; and
at least one instance of anticipated cheating during the anticipated testing activity.

### Claim 20 (depends on 13)

The system of claim 13, wherein the server is further configured, responsive to the anticipated situational risk score exceeding the threshold, to automatically update the database entry to include a recommendation for a change to a format for the exam data including:
increasing a total number of exam questions within an exam bank; and
a modification to a delivery mode for the exam data.