# US 2020/0350057 A1 · Remote computing analysis for cognitive state data metrics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0350057 A1
- Document ID: US20200350057A1
- Status: Published application
- Kind: A1
- Application number: 16/934,069
- Filed: 2020-07-21
- Published: 2020-11-05
- Earliest claimed: 2010-06-07
- Assignees: Affectiva, Inc.
- Applicants: Affectiva, Inc.
- Inventors: Rana el Kaliouby; Rosalind Wright Picard; Richard Scott Sadowsky
- Web page: https://patentgrep.com/patent/US20200350057A1

## CPC

- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/174: {Facial expression recognition}
- G16H 20/70: relating to mental therapies, e.g. psychological therapy or autogenous training
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Remote computing analysis for cognitive state data metrics is performed. Cognitive state data from a plurality of people is collected as they interact with a rendering. The cognitive state data includes video facial data collected on one or more local devices from the plurality of people. Information is uploaded to a remote server. The information includes the cognitive state data. A facial expression metric based on a plurality of image classifiers is calculated for each individual within the plurality of people. Cognitive state information is generated for each individual, based on the facial expression metric for each individual. The cognitive state information for each individual within the plurality of people who interacted with the rendering is aggregated. The aggregation is based on the facial expression metric for each individual. The cognitive state information that was aggregated is displayed on at least one of the one or more local devices.

## Claims

### Claim 1

A computer-implemented method for analysis comprising:
collecting cognitive state data from a plurality of people as they interact with a rendering, wherein the cognitive state data includes video facial data, collected on one or more local devices, from the plurality of people;
uploading information to a remote server, wherein the information includes the cognitive state data;
calculating a facial expression metric, based on a plurality of image classifiers, for each individual within the plurality of people;
generating cognitive state information for each individual, based on the facial expression metric for each individual;
aggregating the cognitive state information for each individual within the plurality of people who interacted with the rendering, based on the facial expression metric for each individual; and
displaying, with the rendering, the cognitive state information that was aggregated, on at least one of the one or more local devices.

### Claim 2 (depends on 1)

The method of claim 1 wherein the cognitive state information includes a cognitive state metric.

### Claim 3 (depends on 1)

The method of claim 1 wherein the aggregated cognitive state information includes aggregated cognitive state metrics.

### Claim 4 (depends on 3)

The method of claim 3 further comprising deriving norms for the plurality of people.

### Claim 5 (depends on 4)

The method of claim 4 wherein the norms for the plurality of people are based on the aggregated cognitive state information.

### Claim 6 (depends on 4)

The method of claim 4 wherein the norms for the plurality of people are based on contextual information.

### Claim 7 (depends on 4)

The method of claim 4 wherein the norms are derived based on cognitive state event temporal signatures.

### Claim 8 (depends on 1)

The method of claim 1 further comprising collecting further cognitive state data from a first individual and comparing the further cognitive state data from the first individual with the aggregated cognitive state information.

### Claim 9 (depends on 1)

The method of claim 1 further comprising associating the cognitive state information that was aggregated with the rendering.

### Claim 10 (depends on 1)

The method of claim 1 further comprising synchronizing the aggregated cognitive state information with the rendering.

### Claim 11 (depends on 1)

The method of claim 1 further comprising capturing contextual information about the rendering.

### Claim 12 (depends on 11)

The method of claim 11 wherein the contextual information includes a timeline, a progression of webpages, or an actigraph.

### Claim 13 (depends on 1)

The method of claim 1 further comprising inferring cognitive states based on the cognitive state data collected from the plurality of people.

### Claim 14 (depends on 1)

The method of claim 1 wherein the calculating includes using an image classifier from a plurality of image classifiers to detect one or more faces in the facial data.

### Claim at position 15

15-16. (canceled)

### Claim 17 (depends on 1)

The method of claim 1 further comprising tracking eyes to identify the rendering with which interacting is accomplished.

### Claim 18 (depends on 17)

The method of claim 17 wherein the tracking of the eyes identifies a portion of the rendering on which the eyes are focused.

### Claim 19 (depends on 18)

The method of claim 18 further comprising recording of eye dwell time on the rendering and associating information on the eye dwell time to the rendering and to the cognitive state data.

### Claim 20 (depends on 1)

The method of claim 1 wherein aggregation of the cognitive state information is performed on a demographic basis so that cognitive state information is grouped based on the demographic basis.

### Claim 21 (depends on 1)

The method of claim 1 wherein the cognitive state information includes two or more cognitive state metrics for an individual within the plurality of people.

### Claim 22 (depends on 21)

The method of claim 21 wherein the aggregating further comprises aggregating the two or more cognitive state metrics for the individual with two or more cognitive state metrics for an additional individual within the plurality of people.

### Claim 23 (depends on 22)

The method of claim 22 wherein the further aggregating is based on a comparison of matched cognitive state metrics for the individual and the additional individual.

### Claim 24 (depends on 1)

The method of claim 1 further comprising creating a visual representation of one or more of the aggregated cognitive state information and cognitive state information on an individual from the plurality of people.

### Claim 25 (depends on 24)

The method of claim 24 wherein the visual representation displays the aggregated cognitive state information on a demographic basis.

### Claim 26

(canceled)

### Claim 27

A computer program product embodied in a non-transitory computer readable medium for analysis, the computer program product comprising code which causes one or more processors to perform operations of:
collecting cognitive state data from a plurality of people as they interact with a rendering, wherein the cognitive state data includes video facial data, collected on one or more local devices, from the plurality of people;
uploading information to a remote server, wherein the information includes the cognitive state data;
calculating a facial expression metric, based on a plurality of image classifiers, for each individual within the plurality of people;
generating cognitive state information for each individual, based on the facial expression metric for each individual;
aggregating the cognitive state information for each individual within the plurality of people who interacted with the rendering, based on facial expression metric for each individual; and
displaying the cognitive state information that was aggregated, with the rendering, on at least one of the one or more local devices.

### Claim 28

A system for analysis comprising:
a memory which stores instructions;
one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
collect cognitive state data from a plurality of people as they interact with a rendering, wherein the cognitive state data includes video facial data, collected on one or more local devices, from the plurality of people;
upload information to a remote server, wherein the information includes the cognitive state data;
calculate a facial expression metric, based on a plurality of image classifiers, for each individual within the plurality of people;
generate cognitive state information for each individual, based on the facial expression metric for each individual;
aggregate the cognitive state information for each individual within the plurality of people who interacted with the rendering, based on facial expression metric for each individual; and
display the cognitive state information that was aggregated, with the rendering, on at least one of the one or more local devices.
