# US 2024/0177524 A1 · System and method for classifying activity of users based on micro-expression and emotion using ai

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177524 A1
- Document ID: US20240177524A1
- Status: Published application
- Kind: A1
- Application number: 18/283,441
- Filed: 2022-03-23
- Published: 2024-05-30
- Earliest claimed: 2021-03-23
- Assignees: Rn Chidakashi Technologies Private Limited
- Applicants: Rn Chidakashi Technologies Private Limited
- Inventors: Prashant Iyengar; Hardik Godara
- Web page: https://patentgrep.com/patent/US20240177524A1

## CPC

- G06Q 10/06398: {Performance of employee with respect to a job function}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06N 20/00: Machine learning
- G06Q 30/06: Buying, selling or leasing transactions
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G09B 5/06: with both visual and audible presentation of the material to be studied
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A system and method for automatically classifying an activity of a user 102 during a proposal by an agent 104 to a user based on micro-expression and emotion of the user that provides a succeeding response to the agent 104 such that the proposal becomes successful using an artificial intelligence model is provided. The system includes a facial micro-expression unit 106, an expression analyser 110, the artificial intelligence model 112. The facial micro-expression unit 106 captures an interactive sequence of audio-visual information. The expression analyser 110 processes the interactive sequence of audio-visual information using the artificial intelligence model to determine an emotion and intensity of emotion of the user. The expression analyser 110 creates a record of a set of questions and responses. The expression analyzer 110 provides the succeeding response to the agent based on the created record using a wearable device 114.

## Claims

### Claim 1

A system (100) for automatically classifying an activity of a user (102) during a proposal by an agent (104) to the user (102) based on micro-expression and emotion of the user (102) using an artificial intelligence model (112), the system (100) comprising:
a facial micro-expression unit (106) that is mounted at a position close to the user (102) to (i) capture a face of the agent (104) and a face of the user (102) appropriately, and (ii) record a voice of the agent (104) and a voice of the user (102), wherein the facial micro-expression unit (106) comprises at least one camera or a microphone to capture an interactive sequence of audio-visual information during the proposal by the agent (104); and
an expression analyzer (110) that acquires the interactive sequence of audio-visual information continuously in real-time from the facial micro-expression unit (106) and process the sequence of the audio-visual information using the artificial intelligence model (112), wherein the expression analyzer (110) comprises,
a memory unit that stores a database and the artificial intelligence model (112); and
a processor (116) that is configured to execute a set of instructions and is configured to,
characterized in that,
train the artificial intelligence model (112) using historical interactive sequences of audio-visual information between the user (102) and the agent (104), historical activities of the user, historical proposals, historical users, historical agents, historical succeeding responses by,
segmenting, using semantic analysis technique, a conversation of the historical interactive sequence of audio-visual information during a historical proposal by analyzing a context of the conversation to determine an exact meaning of words in the conversation based on the context;
determining, using a micro-expression analysis technique, and a gait analysis technique, at least one historical emotion of the user and historical intensity of the at least one emotion;
correlating the historical audio-visual session with the at least one emotion of the user (102), and the intensity of the at least one emotion to classify an activity into at least one of positive events or negative events, wherein the positive events refer to a successful proposal and the negative events refer to an unsuccessful proposal; and
obtaining a trained artificial intelligence model by associating a historical succeeding response of the user (102) during the historical proposal based on the classification of the activity;
determine, using the trained artificial intelligence model (112), the at least one emotion, and the intensity of the at least one emotion of the user (102) during the proposal in real-time; and
providing a successful response to the agent (104) by creating, a record of a set of questions and responses from at least one of (i) the determined at least one emotion, and the determined intensity of the at least one emotion of the user (102) during the proposal in real-time, or (ii) the response is detected if a question is from the standard training record

### Claim 2 (depends on 1)

The system (100) as claimed in claim 1, wherein the processor (116) is configured to detect pairs of dialogue adjacency between the user (102) and the agent (104) to identify the exact meaning of words in the conversation.

### Claim 3 (depends on 1)

The system (100) as claimed in claim 1, wherein the processor (116) is configured to combine at least one answer by the agent (104) from the created record of the set of questions and responses based on the pairs of dialogue adjacency when at least one new question is asked by the user (102).

### Claim 4 (depends on 1)

The system (100) as claimed in claim 1, wherein the processor (116) is configured to retrain the artificial intelligence (112) if there is a misalignment between real-time response and a predicted event for the proposal by the agent (104) to the user (102).

### Claim 5 (depends on 1)

The system (100) as claimed in claim 1, wherein the processor (116) is configured to identify a new set of questions asked by the user (102) in the real-time proposals by the agent (104) to the user (102) for training the artificial intelligence model (112).

### Claim 6 (depends on 1)

The system (100) as claimed in claim 1, wherein the processor (116) is configured to determine at least one muscle point on the face of the user and identifying at least one micro expression of the user (102) and at least one physical movement of the user (102).

### Claim 7 (depends on 1)

The system (100) as claimed in claim 1, wherein the proposal is varied using at least one of a user's identity profile, a gender, age, ethnicity detection, or positive or negative responses from the users to make the proposal successful.

### Claim 8 (depends on 2)

The system (100) as claimed in claim 2, wherein the processor (116) is configured to identify the historical proposal that is successful or unsuccessful related to a group of users for training the agent (104).

### Claim 9 (depends on 1)

The system (100) as claimed in claim 1, wherein the interactive sequence of audio-visual information comprises at least one of a sequence of visual frames captured by the facial micro-expression unit (106), or a sequence of audio frames captured by the facial micro-expression unit (106), wherein the successful response is communicated to the agent (104) through a wearable device (114).

### Claim 10

A processor-implemented method for automatically classifying an activity of a user (102) during a proposal by an agent (104) to the user (102) based on micro-expression and emotion of the user (102) using an artificial intelligence model (112), the method comprising:
capturing a face of the agent (104) and a face of the user (102) appropriately, and recording a voice of the agent (104) and a voice of the user (102) by a facial micro-expression unit (106) that is mounted at a position close to the user (102), wherein the facial micro-expression unit (106) comprises at least one camera or a microphone to capture an interactive sequence of audio-visual information during the proposal by the agent (104);
acquiring, by an expression analyzer (110), the interactive sequence of audio-visual information continuously in real-time from the facial micro-expression unit (106) and processing the sequence of the audio-visual information using the artificial intelligence model 11 (112);
characterized in that,
training the artificial intelligence model (112) using providing historical interactive sequences of audio-visual information between the user (102) and the agent (104), historical activities of the user, historical proposals, historical users, historical agents, historical succeeding responses by,
segmenting, using semantic analysis technique, a conversation of the historical interactive sequence of audio-visual information during a historical proposal by analyzing a context of the conversation to determine an exact meaning of words in the conversation based on the context;
determining, using a micro-expression analysis technique, and a gait analysis technique, at least one historical emotion of the user and historical intensity of the at least one emotion;
correlating the historical audio-visual session with the at least one emotion of the user, and the intensity of the at least one emotion to classify an activity into at least one of positive events or negative events, wherein the positive events refer to a successful proposal and the negative events refer to an unsuccessful proposal; and
obtaining a trained artificial intelligence model by associating a historical succeeding response of the user (102) during the historical proposal based on the classification of the activity;
determining, using the trained artificial intelligence model, the at least one emotion, and the intensity of the at least one emotion of the user (102) during the proposal in real-time;
and providing a successful response to the agent (104) by creating, a record of a set of questions and responses from at least one of (i) the determined at least one emotion, and the determined intensity of the at least one emotion of the user (102) during the proposal in real-time, or (ii) the response is detected if a question is from the standard training record.
