# US 12,118,148 B1 · EMG-based speech detection and communication > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,148 B1 - Document ID: US12118148B1 - Status: Granted - Kind: B1 - Application number: 17/650,805 - Filed: 2022-02-11 - Granted: 2024-10-15 - Earliest claimed: 2021-02-14 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Assif Ziv - Web page: https://patentgrep.com/patent/US12118148B1 ## CPC - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 1/163: {Wearable computers, e.g. on a belt} - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 15/16: using artificial neural networks - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/24: Speech recognition using non-acoustical features - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H) - H04M 1/7243: with interactive means for internal management of messages - H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)} - G10L 13/00: Speech synthesis; Text to speech systems - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G10L 2015/0638: {Interactive procedures} - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - H04M 2201/40: using speech recognition - H04R 1/14: Throat mountings for microphones - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Systems and methods are provided for performing operations comprising: detecting, by one or more electromyograph (EMG) electrodes of an EMG communication device, subthreshold muscle activation signals of one or more muscles associated with speech production, the subthreshold muscle activation signals being generated in response to inner speech of a user; applying a machine learning technique to the subthreshold muscle activation signals to estimate one or more speech features corresponding to the subthreshold muscle activation signals, the machine learning technique being trained to establish a relationship between a plurality of training subthreshold muscle activation signals and ground truth speech features; generating visual or audible output based on the one or more speech features; and causing the visual or audible output to be processed by a messaging application to engage a feature of the messaging application. ## Claims ### Claim 1 A method comprising: detecting a trigger word by a microphone of an EMG communication device; transitioning the EMG communication device between an overt calibration mode to perform overt calibration of a machine learning technique and a covert calibration mode; in response to detecting the trigger word, detecting, by one or more EMG electrodes of the EMG communication device, subthreshold muscle activation signals of one or more muscles associated with speech production, the subthreshold muscle activation signals being generated in response to inner speech of a user; applying the machine learning technique to the subthreshold muscle activation signals to estimate one or more speech features corresponding to the subthreshold muscle activation signals, the machine learning technique being trained to establish a relationship between a plurality of training subthreshold muscle activation signals and ground truth speech features; generating at least one of a visual or audible output based on the one or more speech features; and causing the at least one the visual or audible output to be processed by a messaging application to engage a feature of the messaging application. ### Claim 2 (depends on 1) The method of claim 1, further comprising synthesizing the one or more speech features to generate the audible output, wherein the feature of the messaging application comprises a voice message transmission feature in which the audible output is transmitted from the messaging application of the user to another messaging application of another user. ### Claim 3 (depends on 2) The method of claim 2, further comprising processing the one or more speech features by a neural network to generate the audible output. ### Claim 4 (depends on 3) The method of claim 3, wherein the neural network comprises a WaveNet network. ### Claim 5 (depends on 1) The method of claim 1, further comprising processing the one or more speech features by a neural network classifier to associate the one or more speech features with one or more words or phonemes, wherein the feature of the messaging application comprises a text message transmission feature in which textual representations of the one or more words or phonemes are transmitted from the messaging application of the user to another messaging application of another user. ### Claim 6 (depends on 1) The method of claim 1, wherein the inner speech of the user comprises imaginary speech including a voluntary act of speaking without moving any of the muscles associated with speech production. ### Claim 7 (depends on 1) The method of claim 1, wherein the EMG communication device comprises a wearable collar device, the wearable collar device comprising the one or more EMG electrodes, a microphone, and a communication device, the EMG communication device being in communication with a mobile device that implements the messaging application. ### Claim 8 (depends on 1) The method of claim 1, further comprising training the machine learning technique by performing training operations comprising: computing a deviation between an estimated set of speech features and a first set of ground truth speech features associated with a first set of subthreshold muscle activation signals; and updating one or more parameters of the machine learning technique based on the computed deviation. ### Claim 9 (depends on 1) The method of claim 1, further comprising training the machine learning technique by performing operations comprising: receiving training data comprising a first set of subthreshold muscle activation signals and a first set of ground truth speech features associated with the first set of subthreshold muscle activation signals; processing the first set of subthreshold muscle activation signals by the machine learning technique to generate an estimated set of speech features; computing a deviation between the estimated set of speech features and the first set of ground truth speech features associated with the first set of subthreshold muscle activation signals; and updating one or more parameters of the machine learning technique based on the computed deviation. ### Claim 10 (depends on 1) The method of claim 1, further comprising: setting the EMG communication device into the overt calibration mode to perform overt calibration of the machine learning technique; detecting speech input from a microphone of the EMG communication device in response to setting the EMG communication device into the overt calibration mode; accessing a first set of subthreshold muscle activation signals corresponding to the speech input from the one or more EMG electrodes; converting the speech input to a first set of training speech features; processing the first set of subthreshold muscle activation signals by the machine learning technique to generate a first set of estimated speech features; comparing the first set of estimated speech features with the first set of training speech features; and computing a deviation between the first set of estimated speech features with the first set of training speech features. ### Claim 11 (depends on 10) The method of claim 10, further comprising: determining that the deviation transgresses a calibration completion threshold; and in response to determining that the deviation transgresses the calibration completion threshold, notifying the user that the EMG communication device has completed overt calibration. ### Claim 12 (depends on 10) The method of claim 10, wherein the set of subthreshold muscle activation signals are accessed prior to detecting the speech input from the microphone. ### Claim 13 (depends on 10) The method of claim 10, further comprising: determining that the deviation transgresses a calibration completion threshold; and in response to determining that the deviation transgresses the calibration completion threshold, transitioning the EMG communication device to the covert calibration mode. ### Claim 14 (depends on 13) The method of claim 13, wherein the covert calibration mode performs operations comprising: generating an instruction to the user to perform inner speech corresponding to a stimulus, the stimulus comprising a word or phrase; detecting a second set of subthreshold muscle activation signals by the one or more EMG electrodes in response to generating the instruction; processing the second set of subthreshold muscle activation signals by the machine learning technique to generate a second set of estimated speech features; comparing the second set of estimated speech features with a second set of training speech features corresponding to the word or phrase; and computing a deviation between the second set of estimated speech features with the second set of training speech features. ### Claim 15 (depends on 14) The method of claim 14, further comprising training a neural network to map the first set of subthreshold muscle activation signals to the second set of subthreshold muscle activation signals. ### Claim 16 (depends on 14) The method of claim 14, further comprising detecting speech input from a microphone of the EMG communication device after detecting the second set of subthreshold muscle activation signals, the speech input comprising the speech features corresponding to the word or phrase. ### Claim 17 (depends on 1) The method of claim 1, wherein audio feedback is provided to the user while the subthreshold muscle activation signals are being detected, the audio feedback comprising the audible output generated based on the one or more speech features. ### Claim 18 A system comprising: a storage device of an electromyograph (EMG) communication device; and at least one processor of the EMG communication device configured to perform operations comprising: detecting a trigger word by a microphone of the EMG communication device; transitioning the EMG communication device between an overt calibration mode to perform overt calibration of a machine learning technique and a covert calibration mode; in response to detecting the trigger word, detecting, by one or more EMG electrodes of the EMG communication device, subthreshold muscle activation signals of one or more muscles associated with speech production, the subthreshold muscle activation signals being generated in response to inner speech of a user; applying the machine learning technique to the subthreshold muscle activation signals to estimate one or more speech features corresponding to the subthreshold muscle activation signals, the machine learning technique being trained to establish a relationship between a plurality of training subthreshold muscle activation signals and ground truth speech features; generating visual or audible output based on the one or more speech features; and causing the visual or audible output to be processed by a messaging application to engage a feature of the messaging application. ### Claim 19 A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: detecting a trigger word by a microphone of an EMG communication device; transitioning the EMG communication device between an overt calibration mode to perform overt calibration of a machine learning technique and a covert calibration mode; in response to detecting the trigger word, detecting, by one or more EMG electrodes of the EMG communication device, subthreshold muscle activation signals of one or more muscles associated with speech production, the subthreshold muscle activation signals being generated in response to inner speech of a user; applying the machine learning technique to the subthreshold muscle activation signals to estimate one or more speech features corresponding to the subthreshold muscle activation signals, the machine learning technique being trained to establish a relationship between a plurality of training subthreshold muscle activation signals and ground truth speech features; generating visual or audible output based on the one or more speech features; and causing the visual or audible output to be processed by a messaging application to engage a feature of the messaging application.