# US 2024/0185880 A1 · Systems and Methods for Emotion Detection, Prediction, Annotation, and Coaching

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0185880 A1
- Document ID: US20240185880A1
- Status: Published application
- Kind: A1
- Application number: 18/073,952
- Filed: 2022-12-02
- Published: 2024-06-06
- Earliest claimed: 2022-12-02
- Assignees: Mirro.ai
- Inventors: Maithilee Lalit Samant; Barry Randall Schrag; Rajashree Suresh Varma
- Web page: https://patentgrep.com/patent/US20240185880A1

## CPC

- G10L 25/72: for transmitting results of analysis
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G10L 25/54: for retrieval
- G10L 25/63: for estimating an emotional state
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system that performs emotion detection, prediction, and coaching system is disclosed. The system includes a media feature extractor, which extracts features from different media modalities, a perceiver module which detects or predicts emotional response for a given audience, a coaching module that generates context-based, personalized coaching in the form of commentary, retrospective analysis, tips kudos, events, and scores. The intermediate results, like media feature, perceiver output, and context, are stored in an emotion association database, which can be used as reference data by the system.

## Claims

### Claim 1

A method to improve human communication, comprising:
extracting features from a plurality of media streams;
providing the extracted features to a deep learning machine and predicting an emotional response for a given audience; and
coaching a user to improve human communication based on the predicted emotional response.

### Claim 2 (depends on 1)

The method of claim 1, comprising receiving text, voice, and video stream of a face and detecting emotion from the face.

### Claim 3 (depends on 2)

The method of claim 2, comprising receiving text, voice, and video stream of a body posture.

### Claim 4 (depends on 1)

The method of claim 1, comprising performing commentary/retrospect analysis by annotating a media stream to assign a speaker's emotion and energy level.

### Claim 5 (depends on 1)

The method of claim 1, comprising analyzing voice tone and/or word choice and/or body language

### Claim 6 (depends on 1)

The method of claim 1, wherein the media stream comprises audio, video, or an audio and video stream.

### Claim 7 (depends on 1)

The method of claim 1, comprising pre-recording and uploading the media streams for analysis.

### Claim 8 (depends on 1)

The method of claim 1, comprising analyzing the media streams live through online meetings, media-broadcast, or presentations in real-time.

### Claim 9 (depends on 1)

The method of claim 1, comprising generating emotions, energy, and coaching feedback for subsequent queries, reports, and analytics.

### Claim 10 (depends on 1)

The method of claim 1, comprising capturing live events or online meetings and providing timely interventions with tips based on user emotional states, meeting context, and cultural context.

### Claim 11 (depends on 1)

The method of claim 1, comprising annotating analytics, feedback, or emotions on an input media clip and providing embedded frame-by-frame analysis.

### Claim 12 (depends on 1)

The method of claim 1, comprising generating a coaching feedback and communicating the coaching feedback to the user in real-time through messages, annotations, or events.

### Claim 13 (depends on 1)

The method of claim 1, comprising detecting emotion by a robot and improving robot-human communications

### Claim 14 (depends on 1)

The method of claim 1, comprising coaching a call-center agent with the predicted emotional response.

### Claim 15

A system to improve human communication, comprising:
a processor;
a camera to capture video media stream;
a microphone to capture sound media stream;
a media feature extractor for extracting features from the video and sound media streams;
a perceiver module coupled to the media feature extractor for providing the extracted features to a deep learning machine and predicting an emotional response for a given audience; and
a coaching module coupled to the perceiver module for improving human communication based on the predicted emotional response.

### Claim 16 (depends on 15)

The system of claim 15, comprising, comprising means for detecting facial movements and correlating the facial movements with voice of a human to predict the emotional response.

### Claim 17 (depends on 15)

The system of claim 15, comprising a call center agent computer, comprising means for providing call-center agent with advice to improve agent to human interaction.

### Claim 18 (depends on 15)

The system of claim 15, comprising a vehicular computer, comprising means for providing driver feedback to improve safety based on the predicted emotional response.

### Claim 19 (depends on 15)

The system of claim 15, comprising a robot, comprising means for providing human feedback by the robot based on the predicted emotional response.

### Claim 20 (depends on 15)

The system of claim 15, comprising an emotion association database to store historical media feature, perceiver output, and context as reference data.
