# US 12,119,022 B2 · Cognitive assistant for real-time emotion detection from human speech > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,119,022 B2 - Document ID: US12119022B2 - Status: Granted - Kind: B2 - Application number: 17/536,673 - Filed: 2021-11-29 - Published: 2022-03-17 - Granted: 2024-10-15 - Earliest claimed: 2020-01-21 - Applicants: Rishi Amit Sinha; Ria Sinha - Inventors: Rishi Amit Sinha; Ria Sinha - Web page: https://patentgrep.com/patent/US12119022B2 ## CPC - G06N 20/00: Machine learning - G10L 25/63: for estimating an emotional state - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)} - G10L 21/0208: Noise filtering - G10L 25/30: using neural networks - H04L 67/55: Push-based network services - H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)} - H04R 25/558: {Remote control, e.g. of amplification, frequency} - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2021/065: {Aids for the handicapped in understanding} - H04R 2205/041: Adaptation of stereophonic signal reproduction for the hearing impaired - H04R 2225/43: Signal processing in hearing aids to enhance the speech intelligibility - H04R 2225/55: Communication between hearing aids and external devices via a network for data exchange - H04R 2430/01: Aspects of volume control, not necessarily automatic, in sound systems - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Systems and methods used in a cognitive assistant for detecting human emotions from speech audio signals is described. The system obtains audio signals from an audio receiver and extracts human speech samples. Subsequently, it runs a machine learning based classifier to analyze the human speech signal and classify the emotion observed in it. The user is then notified, based on their preferences, with a summary of the emotion detected. Notifications can also be sent to other systems that have been configured to receive them. Optionally, the system may include the ability to store the speech sample and emotion classification detected for future analysis. The system's machine learning classifier is periodically re-trained based on labelled audio speech data and updated. ## Claims ### Claim 1 A system comprising: an audio receiver; a processing system connected to the audio receiver; a notification system connected to the processing system, wherein the processing system is configured to i) obtain audio signal from the audio receiver; ii) process the audio signal to detect if human speech is present; iii) responsive to the audio signal containing human speech, run a machine learning based classifier to analyze the speech audio signal and output an emotion detected in speech, wherein the detected emotion is one of calm, happy, sad, angry, fearful, surprise, and disgust; v) send the emotion detected to a notification system; vi) loop back to i), wherein the machine learning classifier is periodically trained externally based on labelled audio sample data and updated in the system, and wherein the processing system is further configured to receive feedback from a user that the detected emotion was incorrect or unknown, and process the feedback for the labelled audio sample data. ### Claim 2 (depends on 1) The system of claim 1, wherein the processing system has a filter and an amplifier to output an improved copy of the received audio signal or store it digitally. ### Claim 3 (depends on 1) The system of claim 1, wherein the notification system is a mobile device push notification configured by the user. ### Claim 4 (depends on 1) The system of claim 1, wherein the notification system responds to an API request from an external system. ### Claim 5 (depends on 1) The system of claim 1, wherein notification preferences are configured by the user. ### Claim 6 (depends on 1) The system of claim 1, where the system is running as an application on a mobile device, wherein the audio receiver is a microphone on the mobile device, the processing system is a processor on the mobile device and the notification system is a screen and vibration alerts. ### Claim 7 (depends on 1) The system of claim 1, wherein the audio receiver is a separate device communicatively coupled to the processing system running on a computer. ### Claim 8 A method comprising: i) obtaining audio signal from an audio receiver; ii) processing the audio signal to detect if human speech is present; iii) responsive to the audio signal containing human speech, running a machine learning based classifier to analyze the speech audio signal and output an emotion detected in speech, wherein the detected emotion is one of calm, happy, sad, angry, fearful, surprise, and disgust; v) sending the emotion detected to a notification system; vi) looping back to i), wherein the machine learning classifier is periodically trained externally based on labelled audio sample data and updated in the system, and wherein the method further comprises receiving feedback from a user that the detected emotion was incorrect or unknown, and processing the feedback for the labelled audio sample data. ### Claim 9 (depends on 8) The method of claim 8, further comprising of a filter and an amplifier to output an improved copy of the received audio signal or store it digitally. ### Claim 10 (depends on 8) The method of claim 8, wherein the notification method is a mobile device push notification configured by the user. ### Claim 11 (depends on 8) The method of claim 8, wherein the notification method responds to an API request from an external system. ### Claim 12 (depends on 8) The method of claim 8, wherein the notification method can be configured by the user. ### Claim 13 A non-transitory computer-readable medium comprising instructions that, when executed, cause a processing system to perform steps of: i) obtaining audio signal from an audio receiver; ii) processing the audio signal to detect if human speech is present; iii) responsive to the audio signal containing human speech, running a machine learning based classifier to analyze the speech audio signal and output an emotion detected in speech, wherein the detected emotion is one of calm, happy, sad, angry, fearful, surprise, and disgust; v) sending the emotion detected to a notification system; vi) looping back to i), wherein the machine learning classifier is periodically trained externally based on labelled audio sample data and updated in the system, and wherein the steps further comprise receiving feedback from a user that the detected emotion was incorrect or unknown, and processing the feedback for the labelled audio sample data.