# US 11,908,345 B2 · Engagement level determination and dissemination

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,345 B2
- Document ID: US11908345B2
- Status: Granted
- Kind: B2
- Application number: 16/234,255
- Filed: 2018-12-27
- Published: 2020-04-30
- Granted: 2024-02-20
- Earliest claimed: 2018-12-27
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Nese Alyuz Civitci; Cagri Cagatay Tanriover; Sinem Aslan; Eda Okur Kavil; Asli Arslan Esme; Ergin Genc
- Web page: https://patentgrep.com/patent/US11908345B2

## CPC

- G09B 5/08: providing for individual presentation of information to a plurality of student stations
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q 50/20: Education
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/176: {Dynamic expression}
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Various systems and methods for engagement dissemination. A face detector detects a face in video data. A context filterer determines a student is on-platform and a section type. An appearance monitor selects an emotional and a behavioral classifiers. Emotional and behavioral components are classified based on the detected face. A context-performance monitor selects an emotional and a behavioral classifiers specific to the section type, Emotional and behavioral components are classified based on the log data. A fuser combines the emotional components into an emotional state of the student based on confidence values of the emotional components. The fuser combines the behavioral components a behavioral state of the student based on confidence values of the behavioral components. The user determine an engagement level of the student based on the emotional state and the behavioral state of the student.

## Claims

### Claim 1

An apparatus for engagement dissemination, the apparatus comprising:
a hardware processor; and
memory including instructions, which when executed by the hardware processor, cause the hardware processor to:
detect in video data, a face of a student of a plurality of students in a classroom;
determine that the student is on-platform based on log data;
determine a section type for the student based on the log data, wherein the section type is one of an instructional section type or an assessment section type;
select a first emotional classifier specific to the section type;
classify, using the first emotional classifier, a first emotional component based on the detected face;
select a first behavioral classifier specific to the section type;
classify, using the first behavioral classifier, a first behavioral component based on the detected face;
select a second emotional classifier specific to the section type;
classify, using the second emotional classifier, a second emotional component based on the log data and the section type by using input derived from the log data to extract a set of features, the set of features including at least one of: a number of failed first attempts to a number of questions, a number of hints used within a time period, or an amount of time spent on a first attempt on answering questions in a topic;
select a second behavioral classifier specific to the section type;
classify, using the second behavioral classifier, a second behavioral component based on the log data and the section type;
combine the classified first emotional component and the second emotional component into an emotional state of the student based on confidence values of the first emotional component and the second emotional component;
combine the classified first behavioral component and the second behavioral component into a behavioral state of the student based on confidence values of the first behavioral component and the second behavioral component;
determine an engagement level of the student based on the emotional state of the student and the behavioral state of the student; and
display a student avatar representing the student in a user interface, the user interface including a plurality of student avatars that includes the student avatar, each of the plurality of student avatars representing a respective student in the classroom, wherein each of the plurality of student avatars are colored with an engagement color indicating a respective engagement level of the respective student.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the engagement level is one of engaged, may be engaged, or not engaged.

### Claim 3 (depends on 2)

The apparatus of claim 2, further comprising instructions, which when executed by the hardware processor, cause the hardware processor to:
determine the engagement color based on the engagement level of the student.

### Claim 4 (depends on 3)

The apparatus of claim 3, further comprising instructions, which when executed by the hardware processor, cause the hardware processor to display an intervention alert for the student based on the engagement level of the student.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein to determine the engagement color, the instructions, which when executed by the hardware processor, cause the hardware processor to:
determine a number of engagement levels for a previous time period;
determine a number of engagement level for each possible engagement level of the student for a previous time period;
calculate a ratio using the number of engagement levels and the number of engagement level for each possible engagement level;
determine a largest ratio from the calculated ratios; and
determine the engagement color based on the largest ratio.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the behavior state is one of on-task or off-task.

### Claim 7 (depends on 4)

The apparatus of claim 4, wherein the emotional state is one of satisfied, bored, or confused.

### Claim 8 (depends on 1)

The apparatus of claim 1, further comprising instructions, which when executed by the hardware processor, cause the hardware processor to:
calculate first features from the detected face based on a proper subset of face animation parameters; and
calculate second features from the detected face based on movement of the detected face from previous video data, wherein the first features and the second features are used by the first emotional classifier and the second behavioral classifier.

### Claim 9 (depends on 1)

The apparatus of claim 1, further comprising instructions, which when executed by the hardware processor, cause the hardware processor to:
determine a number of questions accessed in a previous context time period; and
determine a number of questions where a hint associated with a question was accessed in the previous context time period, wherein the number of questions and the number of questions where a hint associated with a question was accessed are used by the first behavioral classifier and the second behavioral classifier.

### Claim 10

A method for engagement dissemination, the method comprising operations performed using an electronic processor, the operations comprising:
detecting, in video data, a face of a student of a plurality of students in a classroom;
determining that the student is on-platform based on log data;
determining a section type for the student based on the log data, wherein the section type is one of an instructional section type or an assessment section type;
selecting, using an appearance monitor (APM), a first emotional classifier specific to the section type;
classifying, using the first emotional classifier, a first emotional component based on the detected face;
selecting, using the APM, a first behavioral classifier specific to the section type;
classifying, using the first behavioral classifier, a first behavioral component based on the detected face;
selecting, using a context-performance monitor (CPM), a second emotional classifier specific to the section type;
classifying, using the second emotional classifier, a second emotional component based on the log data and the section type;
selecting, using the CPM, a second behavioral classifier specific to the section type;
classifying, using the second behavioral classifier, a second behavioral component based on the log data and the section type by using input derived from the log data to extract a set of features, the set of features including at least one of: a number of failed first attempts to a number of questions, a number of hints used within a time period, or an amount of time spent on a first attempt on answering questions in a topic;
combining the classified first emotional component and the second emotional component into an emotional state of the student based on confidence values of the first emotional component and the second emotional component;
combining the classified first behavioral component and the second behavioral component into a behavioral state of the student based on confidence values of the first behavioral component and the second behavioral component;
determining an engagement level of the student based on the emotional state of the student and the behavioral state of the student; and
displaying a student avatar representing the student in a user interface, the user interface including a plurality of student avatars that includes the student avatar, each of the plurality of student avatars representing a respective student in the classroom, wherein each of the plurality of student avatars are colored with an engagement color indicating a respective engagement level of the respective student.

### Claim 11 (depends on 10)

The method of claim 10, wherein the engagement level is one of engaged, may be engaged, or not engaged.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
determining the engagement color based on the engagement level of the student.

### Claim 13 (depends on 12)

The method of claim 12, further comprising displaying an intervention alert for the student based on the engagement level of the student.

### Claim 14 (depends on 12)

The method of claim 12, wherein determining the engagement color comprises:
determining a number of engagement levels for a previous time period;
determining a number of engagement level for each possible engagement level of the student for a previous time period;
calculating a ratio using the number of engagement levels and the number of engagement level for each possible engagement level;
determining a largest ratio from the calculated ratios; and
determining the engagement color based on the largest ratio.

### Claim 15 (depends on 10)

The method of claim 10, wherein the behavior state is one of on-task or off-task.

### Claim 16 (depends on 13)

The method of claim 13, wherein the emotional state is one of satisfied, bored, or confused.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
calculating, using the APM, first features from the detected face based on a proper subset of face animation parameters; and
calculating, using the APM, second features from the detected face based on movement of the detected face from previous video data, wherein the first features and the second features are used by the first emotional classifier and the second behavioral classifier.

### Claim 18 (depends on 10)

The method of claim 10, further comprising:
determining, using the CPM, a number of questions accessed in a previous context time period; and
determining, using the CPM, a number of questions where a hint associated with a question was accessed in the previous context time period, wherein the number of questions and the number of questions where a hint associated with a question was accessed are used by the first behavioral classifier and the second behavioral classifier.

### Claim 19

At least one non-transitory computer-readable medium including instructions for engagement dissemination, the computer-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations comprising:
detecting in video data, a face of a student of a plurality of students in a classroom;
determining that the student is on-platform based on log data;
determining a section type for the student based on the log data, wherein the section type is one of an instructional section type or an assessment section type;
selecting, using an appearance monitor (APM), a first emotional classifier specific to the section type;
classifying, using the first emotional classifier, a first emotional component based on the detected face;
selecting, using the APM, a first behavioral classifier specific to the section type;
classifying, using the first behavioral classifier, a first behavioral component based on the detected face;
selecting, using a context-performance monitor (CPM), a second emotional classifier specific to the section type;
classifying, using the second emotional classifier, a second emotional component based on the log data and the section type by using input derived from the log data to extract a set of features, the set of features including at least one of: a number of failed first attempts to a number of questions, a number of hints used within a time period, or an amount of time spent on a first attempt on answering questions in a topic;
selecting, using the CPM, a second behavioral classifier specific to the section type;
classifying, using the second behavioral classifier, a second behavioral component based on the log data and the section type;
combining the classified first emotional component and the second emotional component into an emotional state of the student based on confidence values of the first emotional component and the second emotional component;
combining the classified first behavioral component and the second behavioral component into a behavioral state of the student based on confidence values of the first behavioral component and the second behavioral component;
determining an engagement level of the student based on the emotional state of the student and the behavioral state of the student; and
displaying a student avatar representing the student in a user interface, the user interface including a plurality of student avatars that includes the student avatar, each of the plurality of student avatars representing a respective student in the classroom, wherein each of the plurality of student avatars are colored with an engagement color indicating a respective engagement level of the respective student.

### Claim 20 (depends on 19)

The at least one non-transitory computer-readable medium of claim 19, wherein the engagement level is one of engaged, may be engaged, or not engaged.

### Claim 21 (depends on 20)

The at least one non-transitory computer-readable medium of claim 20, further comprising:
determining the engagement color based on the engagement level of the student.

### Claim 22 (depends on 21)

The at least one non-transitory computer-readable medium of claim 21, further comprising displaying an intervention alert for the student based on the engagement level of the student.

### Claim 23 (depends on 21)

The at least one non-transitory computer-readable medium of claim 21, wherein determining the engagement color comprises:
determining a number of engagement levels for a previous time period;
determining a number of engagement level for each possible engagement level of the student for a previous time period;
calculating a ratio using the number of engagement levels and the number of engagement level for each possible engagement level;
determining a largest ratio from the calculated ratios; and
determining the engagement color based on the largest ratio.

### Claim 24 (depends on 19)

The at least one non-transitory computer-readable medium of claim 19, wherein the behavior state is one of on-task or off-task.

### Claim 25 (depends on 22)

The at least one non-transitory computer-readable medium of claim 22, wherein the emotional state is one of satisfied, bored, or confused.

### Claim 26 (depends on 1)

The apparatus of claim 1, wherein the log data includes website access logs, system logs, or educational content logs.