# US 11,113,890 B2 · Artificial intelligence enabled mixed reality system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,890 B2
- Document ID: US11113890B2
- Status: Granted
- Kind: B2
- Application number: 16/744,781
- Filed: 2020-01-16
- Published: 2021-05-06
- Granted: 2021-09-07
- Earliest claimed: 2019-11-04
- Assignees: Cognizant Technology Solutions India Pvt. Ltd.
- Applicants: Cognizant Technology Solutions India Pvt. Ltd.
- Inventors: Rajkumar Joseph; Safuvan Puthan Peedika; Arun Muthuraj Vedamanickam; Rajagopal Appakutty; Purwa Rathi
- Web page: https://patentgrep.com/patent/US11113890B2

## CPC

- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/094: Adversarial learning
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 25/63: for estimating an emotional state
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The present invention relates to an artificial intelligence based system and method for moderating interaction between interacting users. The attempt is to improve emotional intelligence of users so that a seasoned response and reaction is observed during interaction, even if situations of conflict arise. The disclosure, thus, provides for a mixed reality glass powered assistant that displays the moderated expressions of a customer to the service provider. For the same, the analytical engine upon determining the negative emotions of customer, transforms the image of customer and adds smile to his face, which is presented to the service provider via a mixed reality glass so that he responds back to the customer in a positive manner.

## Claims

### Claim 1

A system for moderating interaction amongst plurality of users, comprising:
an input module configured to receive image data, speech data and physiological signals of a first user from amongst the plurality of users;
an extraction module configured to extract dynamically varying facial expressions, bodily expressions, and aural characteristics from the received image data, speech data and the physiological signals;
an artificial intelligence engine configured to:
perform context driven emotional assessment on the dynamically varying facial expressions, bodily expressions, and aural characteristics;
apply adjustment to a degree of obtaining normalized emotions in an event the context driven emotional assessment is higher than a predetermined threshold; and
a virtual assistant configured to display normalized emotions of the first user to one or more other users such that a moderated response is generated from the one or more of the plurality of users to the adjusted emotions of the first user.

### Claim 2 (depends on 1)

The system, as claimed in accordance with claim 1, wherein the input module is further configured to receive historical activity of temperament of the first user.

### Claim 3 (depends on 1)

The system, as claimed in accordance with claim 1, further comprising one or more cameras or other image capturing devices that are configured to capture the image of the first user from different angles or different portions of frequency spectrum for transmitting to the input module.

### Claim 4 (depends on 1)

The system, as claimed in accordance with claim 1, further comprising one or more audio sensors or haptic sensors that are configured to capture speech data of the first user along with prosodic features: volume, pitch, speed, strength and tone of speech for transmitting to the input module.

### Claim 5 (depends on 1)

The system, as claimed in accordance with claim 1, further comprising a pulse sensor, a heartbeat sensor, a blood pressure sensor, a respiratory rate sensor, or one or more pace frequency sensor to capture physiological signals of the first user for transmitting to the input module.

### Claim 6 (depends on 1)

The system, as claimed in accordance with claim 1, wherein the extraction module is operable to provide image data and speech data to a 3-dimensional convolutional neural network (CNN) to extract the facial expressions, bodily expressions, and aural characteristics from the image and speech data.

### Claim 7 (depends on 5)

The system, as claimed in accordance with claim 5, wherein the extraction module is further operable to track progress of extraction of the facial expressions, bodily expressions, and aural characteristics based on long short-term memory (LSTM) units.

### Claim 8 (depends on 1)

The system, as claimed in accordance with claim 1, wherein the context driven sentiment is assessed based on:
determination of context of interaction amongst the plurality of users from the dynamically varying facial expressions, bodily expressions, and aural characteristics;
associate a representation of emotional state corresponding to the assessed context based on occurrence of speech elements and emotional tone associated therewith using feed forward neural network model; and
compare the represented emotional state with a predetermined threshold for obtaining emotional assessment score.

### Claim 9 (depends on 1)

The system, as claimed in accordance with claim 1, wherein the normalized emotions refer to emotions other than negative emotions: anger, disgust, anxiety, fatigue or the like.

### Claim 10 (depends on 1)

The system, as claimed in accordance with claim 1, wherein artificial intelligence engine is configured to apply adjustment to the degree of obtaining the normalized emotions by morphing negative emotions of the first user with those of positive emotions using variational autoencoder-generative adversarial network.

### Claim 11 (depends on 1)

The system, as claimed in accordance with claim 1, wherein the virtual assistant comprises of a wearable device configured to be worn by the plurality of users other than the first user to receive and display the normalized emotions of the first user to the plurality of users.

### Claim 12 (depends on 11)

The system, as claimed in accordance with claim 11, wherein the wearable device is a wearable eyewear or a glass partition device, and is configured to select a visualization mode, virtual reality mode, augmented reality mode or mixed reality mode or the combination thereof for display of the normalized emotions.

### Claim 13

A method for moderating interaction amongst plurality of users, comprising:
receiving image data, speech data and physiological signals of a first user from amongst the plurality of users;
extracting dynamically varying facial expressions, bodily expressions, and aural characteristics from the received image data, speech data and the physiological signals;
performing context driven emotional assessment on the dynamically varying facial expressions, bodily expressions, and aural characteristics;
applying adjustment to a degree of obtaining normalized emotions in an event the context driven emotional assessment is higher than a predetermined threshold; and
displaying normalized emotions of the first user to one or more other users equipped with virtual assistant such that a moderated response is generated from the one or more of the plurality of users to the adjusted emotions of the first user.

### Claim 14 (depends on 13)

The method, as claimed in accordance with claim 13, further comprising receiving historical activity of temperament of the first user.

### Claim 15 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the image data is captured from different angles or different portions of frequency spectrum by an image capturing device.

### Claim 16 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the speech data is captured along with prosodic features: volume, pitch, speed, strength and tone of speech.

### Claim 17 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the physiological signals consists measuring of pulse rate, heartbeat, blood pressure, respiratory rate, pace frequency and the like.

### Claim 18 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the facial expressions, bodily expressions, and aural characteristics are extracted using a 3-dimensional convolutional neural network (CNN) model.

### Claim 19 (depends on 13)

The method, as claimed in accordance with claim 13, further comprising tracking of progress of extraction of the facial expressions, bodily expressions, and aural characteristics based on long short-term memory (LSTM) units.

### Claim 20 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the context driven sentiment assessment is performed by:
determining context of interaction amongst the plurality of users from the dynamically varying facial expressions, bodily expressions, and aural characteristics;
associating a representation of emotional state corresponding to the assessed context based on occurrence of speech elements and emotional tone associated therewith using feed forward neural network model; and
comparing the represented emotional state with a predetermined threshold for obtaining emotional assessment score.

### Claim 21 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the normalized emotions refer to emotions other than negative emotions: anger, disgust, anxiety, fatigue or the like.

### Claim 22 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the normalized emotions are obtained by morphing negative emotions of the first user with those of positive emotions using variational autoencoder-generative adversarial network.

### Claim 23 (depends on 13)

The method, as claimed in accordance with claim 13, wherein the virtual assistant comprises of a wearable device configured to be worn by the plurality of users other than the first user to receive and display the normalized emotions of the first user to the plurality of users.