# US 2017/0256172 A1 · Student data-to-insight-to-action-to-learning analytics system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0256172 A1
- Document ID: US20170256172A1
- Status: Published application
- Kind: A1
- Application number: 15/451,147
- Filed: 2017-03-06
- Published: 2017-09-07
- Earliest claimed: 2016-03-04
- Assignees: Civitas Learning, Inc.
- Applicants: Civitas Learning, Inc.
- Inventors: David H. Kil; Kyle Derr; Mark Whitfield; Grace Eads; John M. Daly; Clayton Gallaway; Jorgen Harmse; Daya Chinthana Wimalasuriya
- Web page: https://patentgrep.com/patent/US20170256172A1

## CPC

- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G06Q 10/06393: {Score-carding, benchmarking or key performance indicator [KPI] analysis}
- G06Q 50/205: {Education administration or guidance}
- G09B 5/00: Electrically-operated educational appliances (working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F)
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Student data-to-insight-to-action-to-learning analytics system and method use an evidence-based action knowledge database to compute student success predictions, student engagement predictions, and student impact predictions to interventions. The evidence-based action knowledge database is updated by executing a multi-tier impact analysis on impact results of applied interventions. The multi-tier impact analysis includes using changes in key performance indicators (KPIs) for pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

## Claims

### Claim 1

A student data-to-insight-to-action-to-learning analytics method comprising:
computing student success predictions, student engagement predictions, and student impact predictions to interventions using at least linked-event features from multiple student event data sources and an evidence-based action knowledge database, the linked-event features including student characteristic factors that are relevant to student success;
applying appropriate interventions to pilot students when engagement rules are triggered, the engagement rules being based on at least the linked-event features and multi-modal student success prediction scores; and
executing a multi-tier impact analysis on impact results of the applied interventions to update the evidence-based action knowledge database, the multi-tier impact analysis including using changes in key performance indicators (KPIs) for the pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

### Claim 2 (depends on 1)

The method of claim 1, wherein executing a multi-tier impact analysis includes computing utility scores for triggered engagement rule-intervention pairs by looking at changes in KPIs within a defined time window.

### Claim 3 (depends on 2)

The method of claim 2, wherein executing a multi-tier impact analysis further includes determining whether the interventions are message nudges, and for the message nudges, performing natural language processing on the contents of the message nudges to learn characteristics of effective and ineffective messages.

### Claim 4 (depends on 1)

The method of claim 1, wherein applying the appropriate interventions to the pilot students further comprises:
monitoring incoming streams of event data to detect when any of the engagement rules are triggered;
if more than one engagement rule is triggered, prioritizing the engagement rules that are triggered based corresponding utility scores and intersection with recently triggered engagement rules to derive a highest ranked engagement rule; and
applying an intervention that correspond to the highest ranked engagement rule to at least one pilot student.

### Claim 5 (depends on 3)

The method of claim 3, wherein executing a multi-tier impact analysis includes:
aligning the applied interventions with respect to time;
creating a pool of control students that are similar to each pilot student exposed to one of the interventions;
creating groups of pilot and control students that have similar metrics; and
performing difference-of-difference analysis on each applied intervention for the groups of pilot and control students to produce success metrics for cells of CPT.

### Claim 6 (depends on 5)

The method of claim 5, wherein executing a multi-tier impact analysis further includes correlating the success metrics with the utility scores.

### Claim 7 (depends on 6)

The method of claim 6, wherein executing a multi-tier impact analysis includes:
segmenting the students using data footprint;
selecting features and academic terms for matching:
building predictive and propensity-score models for each student-success metric and intervention program to produce prediction scores and propensity scores;
performing a matching process on the pilot and control students to ensure that the pilot and control students are indistinguishable in a statistical sense; and
executing a statistical hypothesis testing to determine if observed difference in student success rates between the pilot and control students is statistically significant.

### Claim 8

A computer-readable storage medium containing program instructions for student data-to-insight-to-action-to-learning analytics method, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:
computing student success predictions, student engagement predictions, and student impact predictions to interventions using at least linked-event features from multiple student event data sources and an evidence-based action knowledge database, the linked-event features including student characteristic factors that are relevant to student success;
applying appropriate interventions to pilot students when engagement rules are triggered, the engagement rules being based on at least the linked-event features and multi-modal student success prediction scores; and
executing a multi-tier impact analysis on impact results of the applied interventions to update the evidence-based action knowledge database, the multi-tier impact analysis including using changes in key performance indicators (KPIs) for the pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

### Claim 9 (depends on 8)

The computer-readable storage medium of claim 8, wherein executing a multi-tier impact analysis includes computing utility scores for triggered engagement rule-intervention pairs by looking at changes in KPIs within a defined time window.

### Claim 10 (depends on 9)

The computer-readable storage medium of claim 9, wherein executing a multi-tier impact analysis further includes determining whether the interventions are message nudges, and for the message nudges, performing natural language processing on the contents of the message nudges to learn characteristics of effective and ineffective messages.

### Claim 11 (depends on 8)

The computer-readable storage medium of claim 8, wherein applying the appropriate interventions to the pilot students further comprises:
monitoring incoming streams of event data to detect when any of the engagement rules are triggered;
if more than one engagement rule is triggered, prioritizing the engagement rules that are triggered based corresponding utility scores and intersection with recently triggered engagement rules to derive a highest ranked engagement rule; and
applying an intervention that correspond to the highest ranked engagement rule to at least one pilot student.

### Claim 12 (depends on 11)

The computer-readable storage medium of claim 11, wherein executing a multi-tier impact analysis includes:
aligning the applied interventions with respect to time;
creating a pool of control students that are similar to each pilot student exposed to one of the interventions;
creating groups of pilot and control students that have similar metrics; and
performing difference-of-difference analysis on each applied intervention for the groups of pilot and control students to produce success metrics for cells of CPT.

### Claim 13 (depends on 12)

The computer-readable storage medium of claim 12, wherein executing a multi-tier impact analysis further includes correlating the success metrics with the utility scores.

### Claim 14 (depends on 13)

The computer-readable storage medium of claim 13, wherein executing a multi-tier impact analysis includes:
segmenting the students using data footprint;
selecting features and academic terms for matching:
building predictive and propensity-score models for each student-success metric and intervention program to produce prediction scores and propensity scores;
performing a matching process on the pilot and control students to ensure that the pilot and control students are indistinguishable in a statistical sense; and
executing a statistical hypothesis testing to determine if observed difference in student success rates between the pilot and control students is statistically significant.

### Claim 15

A student data-to-insight-to-action-to-learning analytics system comprising:
memory;
a processor configured to:
compute student success predictions, student engagement predictions, and student impact predictions to interventions using at least linked-event features from multiple student event data sources and an evidence-based action knowledge database, the linked-event features including student characteristic factors that are relevant to student success;
apply appropriate interventions to pilot students when engagement rules are triggered, the engagement rules being based on at least the linked-event features and multi-modal student success prediction scores; and
execute a multi-tier impact analysis on impact results of the applied interventions to update the evidence-based action knowledge database, the multi-tier impact analysis including using changes in key performance indicators (KPIs) for the pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

### Claim 16 (depends on 15)

The system of claim 15, wherein the processor is configured to compute utility scores for triggered engagement rule-intervention pairs by looking at changes in KPIs within a defined time window to execute the multi-tier impact analysis.

### Claim 17 (depends on 16)

The system of claim 16, wherein the processor is configured to determine whether the interventions are message nudges, and for the message nudges, perform natural language processing on the contents of the message nudges to learn characteristics of effective and ineffective messages to execute the multi-tier impact analysis.

### Claim 18 (depends on 15)

The system of claim 15, wherein the processor is configured to:
monitor incoming streams of event data to detect when any of the engagement rules are triggered;
if more than one engagement rule is triggered, prioritize the engagement rules that are triggered based corresponding utility scores and intersection with recently triggered engagement rules to derive a highest ranked engagement rule; and
apply an intervention that correspond to the highest ranked engagement rule to at least one pilot student.

### Claim 19 (depends on 18)

The system of claim 18, wherein the processor is configured to:
align the applied interventions with respect to time;
create a pool of control students that are similar to each pilot student exposed to one of the interventions;
create groups of pilot and control students that have similar metrics; and
perform difference-of-difference analysis on each applied intervention for the groups of pilot and control students to produce success metrics for cells of CPT.

### Claim 20 (depends on 19)

The system of claim 19, wherein the processor is configured to:
segment the students using data footprint;
selecting features and academic terms for matching:
build predictive and propensity-score models for each student-success metric and intervention program to produce prediction scores and propensity scores;
perform a matching process on the pilot and control students to ensure that the pilot and control students are indistinguishable in a statistical sense; and
execute a statistical hypothesis testing to determine if observed difference in student success rates between the pilot and control students is statistically significant.
