# US 12,602,112 B2 · Determining mental states based on biometric data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,112 B2 - Document ID: US12602112B2 - Status: Granted - Kind: B2 - Application number: 18/121,350 - Filed: 2023-03-14 - Published: 2023-07-06 - Granted: 2026-04-14 - Earliest claimed: 2020-12-15 - Assignees: Neurable Inc. - Applicants: Neurable, Inc. - Inventors: Ramses Eduardo Alcaide; David Stanley; Dereck Padden; James Hamet; Adam Molnar; Jamie Alders; Jegan Candassamy; Arjun Daniel Srinivas - Web page: https://patentgrep.com/patent/US12602112B2 ## CPC - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - A61B 5/1123: {Discriminating type of movement, e.g. walking or running (A61B5/1116, A61B5/112 take precedence)} - A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety} - A61B 5/28: for electrocardiography [ECG] - A61B 5/291: for electroencephalography [EEG] - A61B 5/296: for electromyography [EMG] - A61B 5/369: Electroencephalography [EEG] - A61B 5/375: using biofeedback - A61B 5/389: Electromyography [EMG] - A61B 5/398: Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG] - A61B 5/486: {Biofeedback (using electroencephalography [EEG] A61B5/375)} - A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles} - A61B 5/7267: {involving training the classification device} - A61B 5/742: {using visual displays (displays for heart-related electrical signals, e.g. ECG, A61B5/339)} - G01R 19/0084: {Measuring voltage only} - G05B 13/0265: {the criterion being a learning criterion} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications} - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G10L 25/63: for estimating an emotional state - G16H 40/63: for local operation - G16H 40/67: for remote operation - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor - H04R 1/1008: {Earpieces of the supra-aural or circum-aural type} - H04R 1/1041: {Mechanical or electronic switches, or control elements (switches in general H01H)} - H04R 1/1091: {Details not provided for in groups H04R1/1008 - H04R1/1083} - A61B 5/0205: Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition - A61B 5/1107: {Measuring contraction of parts of the body, e.g. organ or muscle} - A61B 5/168: {Evaluating attention deficit, hyperactivity} - A61B 5/332: Portable devices specially adapted therefor - A61B 5/7221: {Determining signal validity, reliability or quality (preventing, reducing or removing noise induced by motion artefacts A61B5/7207; noise originating from a therapeutic or surgical apparatus A61B5/7217)} - G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns - G16H 20/30: relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising - G16H 20/70: relating to mental therapies, e.g. psychological therapy or autogenous training - H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Analytics Engine that receives one more signal files that include neural signal data of a user based on voltages detected by one or more electrodes on a set of headphones worn by a user. The Analytics Engine preprocesses the data, extracts features from the received data, and feeds the extracted features into one or more machine learning models to generate determined output that corresponds to at least one of a current mental state of the user and a type of facial gesture performed by the user. The Analytics Engine sends the determined output to a computing device to perform an action based on the determined output. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving one or more signal files of data based on voltages detected by one or more electrodes on a set of headphones worn by a user, wherein at least one of the electrodes comprises a first electrode for detecting both an EEG signal and an EMG signal; extracting features from the received data and feeding the extracted features into one or more machine learning models to generate a determined output that corresponds to at least one of a current mental state of the user and a type of one or more facial gestures performed by the user; and sending the determined output to a computing device to perform an action based on the determined output; wherein extracting features comprises: extracting one or more EEG features from the received data based at least on EEG signals detected by the first electrode; feeding at least one of the extracted EEG features into at least one of a first machine learning model for determining a current level of focus of the user, a second machine learning model for determining a current level of attentiveness of the user and a third machine learning model for determining a current level of cognitive load of the user; extracting EMG features from the received data based at least on EMG signals detected by the first electrode; and feeding one or more of the extracted EMG features into a facial gesture machine learning model for determining a type of facial gesture from a plurality of types of facial gestures. ### Claim 2 (depends on 1) The computer-implemented method as in claim 1, wherein each of the electrodes on the set of headphones being situated on a respective ear cuff cushion of the set of headphones. ### Claim 3 (depends on 2) The computer-implemented method as in claim 2, wherein each of the electrodes makes electrical contact to the user via a conductive textile of the set of headphones. ### Claim 4 (depends on 1) The computer-implemented method as in claim 1, wherein the determined output comprises at least one input command, wherein the input command includes at least one of, (i) one or more instructions to initiate at least one of a notification and (ii) one or more instructions for one of (a) a reduction, (b) a minimization and (c) an elimination of presentation of one or more notifications on the computing device; and sending the input command to the computing device. ### Claim 5 (depends on 1) The computer-implemented method as in claim 1, wherein sending the determined output to a computing device to perform an action based on the determined output comprises: detecting a relationship between a current level of a particular type of mental state of the user with respect to a threshold for the particular type of mental state; identifying at least one input command that corresponds to the detected relationship between the current level of the particular type of mental state of the user and the threshold; and sending the input command to the computing device. ### Claim 6 (depends on 5) The computer-implemented method as in claim 5, further comprising: wherein the detected relationship between the current level of the particular type of mental state of the user and the threshold comprises: an indication of the current level of focus of the user being higher than a focus threshold; and wherein the input command comprises a trigger to initiate a reduction of presentation of one or more notifications on the computing device to assist the user in maintaining the current level of high focus. ### Claim 7 (depends on 6) The computer-implemented method as in claim 6, wherein the input command further comprises a trigger to initiate presentation of a color on a portion of the set of headphones to signal to any individual possibly near the user to not disturb the user during the current level of high focus. ### Claim 8 (depends on 5) The computer-implemented method as in claim 5, further comprising: wherein the detected relationship between the current level of the particular type of mental state of the user and the threshold comprises: an indication of the current level of mental fatigue of the user being higher than a mental fatigue threshold; and wherein the input command comprises a trigger to initiate presentation of one or more notifications to prompt the user to perform at least one physical action to assist the user in alleviating the current level of mental fatigue. ### Claim 9 (depends on 1) The computer-implemented method as in claim 1, wherein the type of facial gesture includes a sequence of facial gestures represented in the one or more of the extracted EMG features, the sequence of facial gestures comprising an attempt to match a passcode; and wherein sending the determined output to a computing device to perform an action based on the determined output comprises: determining that the sequence of facial gestures matches the passcode; and based on the sequence of facial gestures matching the passcode, sending output to the computing device indicating an instance of verification of an identity associated with the passcode. ### Claim 10 (depends on 9) The computer-implemented method as in claim 9, wherein the type of facial gesture has one or more corresponding additional components; wherein receiving one or more signal files comprises: receiving one or more signal files of data of one or more additional data type portions; wherein extracting EMG features from the received data comprises: extracting additional features from the one or more additional data type portions; feeding one or more of the extracted EMG features into a facial gesture machine learning model comprises: feeding the extracted additional features into the facial gesture machine learning model to generate a determined output that corresponds to the type of facial gesture performed by the user. ### Claim 11 (depends on 10) The computer-implemented method as in claim 10, comprises: wherein the one or more additional data type portions comprises audio data based on sound detected by one or more microphones associated with the headphones; and wherein the additional features comprise audio features. ### Claim 12 (depends on 10) The computer-implemented method as in claim 10, comprises: wherein the one or more additional data type portions comprises movement data based on at least one change in a physical orientation of a motion detector associated with the set of headphones; wherein the additional features comprise movement features; and wherein the determined output corresponds to the type of facial gesture and a type of head movement. ### Claim 13 (depends on 9) The computer-implemented method as in claim 9, wherein sending the determined output to a computing device to perform an action based on the determined output comprises: identifying an interaction that maps to the determined type of facial gesture; identifying an input command that corresponds to the interaction; and sending the input command to the computing device. ### Claim 14 (depends on 13) The computer-implemented method as in claim 13, wherein the interaction comprises a prompt interaction from the user to the computing device; and wherein the input command comprises a trigger for initiation of a computing device mode in response to the prompt interaction from the user. ### Claim 15 (depends on 14) The computer-implemented method as in claim 14, wherein the prompt interaction comprises one of a request to enter into a wake mode and a request to enter into a sleep mode; and wherein the input command comprises one of a trigger for the computing device to enter into an awakened mode to monitor for an occurrence of at least one subsequent input command and a trigger for the computing device to enter into the sleep mode. ### Claim 16 (depends on 1) The computer-implemented method as in claim 1, further comprising: wherein receiving data based on voltages detected by one or more electrodes on a set of headphones worn by the user comprises receiving data based on voltages generated in response to detection one or more jaw movements; wherein extracting features comprises: extracting features from the received data to generate determined output that corresponds to the one or more jaw movements; and wherein sending the determined output to the computing device comprises: sending a command to the computing device to perform an input command pre-defined as being selected via the one or more jaw movements of the user detected by the one or more electrodes on the set of headphones worn by the user. ### Claim 17 (depends on 1) The computer-implemented method as in claim 1, further comprising: prior to extracting features from the received data: detecting whether one or more signal files of data from a respective electrode channel represent good channel data or bad channel data based on at least one or more impedance criteria; and based on said determination, removing the bad channel data and replacing the removed bad channel data with good data to create one or more improved channels, and wherein extracting features from the received data comprises: extracting features from at least one of the good channels and at least one of the improved channels. ### Claim 18 A system comprising: one or more processors; and a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to: receive one or more signal files of data based on voltages detected by one or more electrodes on a set of headphones worn by a user, wherein at least one of the electrodes comprises a first electrode for detecting both an EEG signal and an EMG signal; extract features from the received data and feeding the extracted features into one or more machine learning models to generate a determined output that corresponds to at least one of a current mental state of the user and a type of one or more facial gestures performed by the user; and send the determined output to a computing device to perform an action based on the determined output; wherein extract features comprises: extract one or more EEG features from the received data based at least on EEG signals detected by the first electrode; feed at least one of the extracted EEG features into at least one of a first machine learning model for determining a current level of focus of the user, a second machine learning model for determining a current level of attentiveness of the user and a third machine learning model for determining a current level of cognitive load of the user; extract EMG features from the received data based at least on EMG signals detected by the first electrode; and feed one or more of the extracted EMG features into a facial gesture machine learning model for determining a type of facial gesture from a plurality of types of facial gestures. ### Claim 19 (depends on 18) The system of claim 18, wherein receiving data based on voltages detected by one or more electrodes on a set of headphones worn by the user comprises receiving data based on voltages generated in response to detection one or more jaw movements; wherein extracting features comprises: extracting features from the received data to generate determined output that corresponds to the one or more jaw movements; and wherein sending the determined output to the computing device comprises: sending a command to the computing device to perform an input command pre-defined as being selected via the one or more jaw movements of the user detected by the one or more electrodes on the set of headphones worn by the user. ### Claim 20 A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to: receive one or more signal files of data based on voltages detected by one or more electrodes on a set of headphones worn by a user, wherein at least one of the electrodes comprises a first electrode for detecting both an EEG signal and an EMG signal; extract features from the received data and feeding the extracted features into one or more machine learning models to generate a determined output that corresponds to at least one of a current mental state of the user and a type of one or more facial gestures performed by the user; and send the determined output to a computing device to perform an action based on the determined output; wherein extract features comprises: extract one or more EEG features from the received data based at least on EEG signals detected by the first electrode; feed at least one of the extracted EEG features into at least one of a first machine learning model for determining a current level of focus of the user, a second machine learning model for determining a current level of attentiveness of the user and a third machine learning model for determining a current level of cognitive load of the user; extract EMG features from the received data based at least on EMG signals detected by the first electrode; and feed one or more of the extracted EMG features into a facial gesture machine learning model for determining a type of facial gesture from a plurality of types of facial gestures.