# US 10,869,626 B2 · Image analysis for emotional metric evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,869,626 B2
- Document ID: US10869626B2
- Status: Granted
- Kind: B2
- Application number: 16/017,037
- Filed: 2018-06-25
- Published: 2018-10-25
- Granted: 2020-12-22
- Earliest claimed: 2010-06-07
- Assignees: Affectiva, Inc.
- Applicants: Affectiva, Inc.
- Inventors: Jason Krupat; Rana el Kaliouby; Jason Radice; Chilton Lyons Cabot
- Web page: https://patentgrep.com/patent/US10869626B2

## CPC

- G06Q 30/0271: {Personalized advertisement}
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/015: {By temperature mapping of body part}
- A61B 5/1176: {Recognition of faces}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/18: for vehicle drivers {or machine operators}
- A61B 5/4833: {Assessment of subject's compliance to treatment}
- A61B 5/4842: {Monitoring progression or stage of a disease}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- A61B 5/7275: {Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor}
- G06Q 10/10: Office automation; Time management
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06Q 30/0282: Rating or review of business operators or products
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/67: for remote operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B 3/113: for determining or recording eye movement
- A61B 5/02055: {Simultaneously evaluating both cardiovascular condition and temperature}
- A61B 5/02405: {Determining heart rate variability}
- A61B 5/0533: Measuring galvanic skin response
- A61B 5/08: Measuring devices for evaluating the respiratory organs
- A61B 5/11: Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb {(A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721)}
- A61B 5/6824: {Arm or wrist}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Techniques are described for image analysis and representation for emotional metric threshold generation. A client device is used to collect image data of a user interacting with a media presentation, where the image data includes facial images of the user. One or more processors are used to analyze the image data to extract emotional content of the facial images. One or more emotional intensity metrics are determined based on the emotional content. The one or more emotional intensity metrics are stored into a digital storage component. The one or more emotional intensity metrics, obtained from the digital storage component, are coalesced into a summary emotional intensity metric. The summary emotional intensity metric is represented.

## Claims

### Claim 1

A computer-implemented method for image analysis comprising:
collecting, at a client device, image data of a user interacting with a media presentation, wherein the image data comprises facial images of the user;
analyzing, using one or more processors, the image data to extract emotional content of the facial images;
determining one or more emotional intensity metrics based on the emotional content; storing the one or more emotional intensity metrics into a digital storage component;
detecting that a threshold value has been met by the one or more emotional intensity metrics, wherein the threshold value is based on evaluation of facial movements of the user;
coalescing the one or more emotional intensity metrics, obtained from the digital storage component, into a summary emotional intensity metric;
comparing the summary emotional intensity metric to an emotional intensity goal, wherein the emotional intensity goal includes increasing a daily percentage of a particular emotion; and
representing the summary emotional intensity metric.

### Claim 2 (depends on 1)

The method of claim 1 wherein the threshold value is based on evaluation of facial movements of a group of users.

### Claim 3 (depends on 1)

The method of claim 1 further comprising generating a video, using the image data of the user, based on the threshold value having been met.

### Claim 4 (depends on 3)

The method of claim 3 further comprising transmitting the video to a display device.

### Claim 5 (depends on 3)

The method of claim 3 wherein the video includes image frames from before the threshold value was met.

### Claim 6 (depends on 1)

The method of claim 1 wherein the representing includes displaying the summary emotional intensity metric or the one or more emotional intensity metrics.

### Claim 7 (depends on 1)

The method of claim 1 further comprising displaying information on content of the media presentation.

### Claim 8 (depends on 1)

The method of claim 1 further comprising augmenting the collecting image data with collecting audio data.

### Claim 9 (depends on 8)

The method of claim 8 wherein the audio data includes speech.

### Claim 10 (depends on 1)

The method of claim 1 further comprising selecting an image from the facial images, wherein the image that was selected is based on a highest intensity emotion.

### Claim 11 (depends on 1)

The method of claim 1 further comprising capturing self-images of the user based on the one or more emotional intensity metrics.

### Claim 12 (depends on 11)

The method of claim 11 further comprising displaying the self-images along with the summary emotional intensity metric.

### Claim 13 (depends on 1)

The method of claim 1 wherein the coalescing further comprises counting occurrences of a specific emotion type within the emotional content.

### Claim 14

A computer-implemented method for image analysis comprising:
collecting, at a client device, image data of a user interacting with a media presentation, wherein the Image data comprises facial images of the user: analyzing, using one or more processors, the image data to extract emotional content of the facial images;
determining one or more emotional intensity metrics based on the emotional content;
storing the one or more emotional intensity metrics into a digital storage component;
detecting that a threshold value has been met by the one or more emotional intensity metrics, wherein the threshold value is based on evaluation of facial movements of the user;
comparing at least one of the one or more emotional intensity metrics to an emotional intensity goal, wherein the emotional intensity goal includes increasing a daily percentage of a particular emotion;
generating a graphical representation of a facial expression for the user based on the threshold value having been met; and
attaching the graphical representation to a representation of the media presentation.

### Claim 15 (depends on 14)

The method of claim 14 wherein the graphical representation includes an emoji or an emoticon.

### Claim 16 (depends on 15)

The method of claim 15 wherein the emoji includes an animated emoji.

### Claim 17 (depends on 14)

The method of claim 14 wherein the threshold value is based on evaluation of facial movements of a group of users.

### Claim 18 (depends on 14)

The method of claim 14 wherein the graphical representation comprises a video.

### Claim 19 (depends on 18)

The method of claim 18 wherein the video includes image frames from before the threshold value was met.

### Claim 20

A computer program product embodied in a non-transitory computer readable medium for image analysis, the computer program product comprising code which causes one or more processors to perform operations of:
collecting, at a client device, image data of a user interacting with a media presentation, wherein the image data comprises facial images of the user;
analyzing, using one or more processors, the image data to extract emotional content of the facial images;
determining one or more emotional intensity metrics based on the emotional content;
storing the one or more emotional intensity metrics into a digital storage component;
detecting that a threshold value has been met by the one or more emotional intensity metrics, wherein the threshold value is based on evaluation of facial movements of the user;
coalescing the one or more emotional intensity metrics, obtained from the storage component, into a summary emotional intensity metric;
comparing the summary emotional intensity metric to an emotional intensity goal, wherein the emotional intensity goal includes increasing a daily percentage of a particular emotion; and
representing the summary emotional intensity metric.

### Claim 21 (depends on 1)

The method of claim 1 wherein the increasing a daily percentage of a particular emotion is for a current time period compared to a summary emotional intensity metric generated based on a previous time period.

### Claim 22 (depends on 21)

The method of claim 21 wherein the particular emotion is happiness and the current time period is a day.

### Claim 23 (depends on 1)

The method of claim 1 further comprising displaying a dashboard providing simultaneous indications of progress toward at least two emotional intensity goals for a current time period.

### Claim 24 (depends on 23)

The method of claim 23 further comprising providing suggestions for activities selected to help the user improve their mood.