# US 2019/0172458 A1 · Speech analysis for cross-language mental state identification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172458 A1 - Document ID: US20190172458A1 - Status: Published application - Kind: A1 - Application number: 16/206,135 - Filed: 2018-11-30 - Published: 2019-06-06 - Earliest claimed: 2017-12-01 - Assignees: Affectiva, Inc. - Applicants: Affectiva, Inc. - Inventors: Taniya Mishra; Islam Faisal; Mohamed Ezzeldin Abdelmonem Ahmed Mohamed - Web page: https://patentgrep.com/patent/US20190172458A1 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/24133: {Distances to prototypes} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 40/174: {Facial expression recognition} - G10L 15/005: {Language recognition} - G10L 15/063: {Training} - G10L 15/16: using artificial neural networks - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 25/63: for estimating an emotional state - G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08) - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2015/223: {Execution procedure of a spoken command} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Techniques are described for speech analysis for cross-language mental state identification. A first group of utterances in a first language is collected, on a computing device, with an associated first set of mental states. The first group of utterances and the associated first set of mental states are stored on an electronic storage device. A machine learning system is trained using the first group of utterances and the associated first set of mental states that were stored. A second group of utterances from a second language is processed, on the machine learning system that was trained, wherein the processing determines a second set of mental states corresponding to the second group of utterances. The second set of mental states is output. A series of heuristics is output, based on the correspondence between the first group of utterances and the associated first set of mental states. ## Claims ### Claim 1 A computer-implemented method for speech analysis comprising: collecting, on a computing device, a first group of utterances in a first language with an associated first set of mental states; storing, on an electronic storage device, the first group of utterances and the associated first set of mental states; training a machine learning system using the first group of utterances and the associated first set of mental states that were stored; processing, on the machine learning system that was trained, a second group of utterances from a second language, wherein the processing determines a second set of mental states corresponding to the second group of utterances; and outputting the second set of mental states. ### Claim 2 (depends on 1) The method of claim 1 further comprising learning, from the first group of utterances and associated first set of mental states and the second group of utterances and associated second set of mental states, to facilitate determining an associated third set of mental states from a third group of utterances. ### Claim 3 (depends on 1) The method of claim 1 further comprising outputting a series of heuristics, based on correspondence between the first group of utterances and the associated first set of mental states. ### Claim 4 (depends on 1) The method of claim 1 wherein the machine learning system includes a deep learning system. ### Claim 5 (depends on 4) The method of claim 4 wherein the machine learning system performs convolving. ### Claim 6 (depends on 1) The method of claim 1 wherein the machine learning system includes a convolutional neural network. ### Claim 7 (depends on 1) The method of claim 1 wherein the first language and the second language are substantially similar. ### Claim 8 (depends on 7) The method of claim 7 wherein the first language and the second language are identical. ### Claim 9 (depends on 1) The method of claim 1 wherein the first language and the second language are different. ### Claim 10 (depends on 1) The method of claim 1 further comprising refining the training of the machine learning system based on one or more additional groups of utterances in the first language or the second language. ### Claim 11 (depends on 1) The method of claim 1 wherein the training comprises segmenting silence from speech in the second group of utterances. ### Claim 12 (depends on 1) The method of claim 1 wherein the training comprises extracting low-level acoustic descriptors from short, overlapping speech segments from the second group of utterances. ### Claim 13 (depends on 12) The method of claim 12 further comprising applying statistical functions to resolve low-level acoustic descriptors over longer speech segments. ### Claim 14 (depends on 12) The method of claim 12 further comprising extracting contextual information from neighboring speech segments. ### Claim 15 (depends on 12) The method of claim 12 further comprising feeding extracted features to a classifier for determining mental states. ### Claim 16 (depends on 1) The method of claim 1 wherein the training comprises estimating mental state metrics over successive, overlapped speech segments. ### Claim 17 (depends on 16) The method of claim 16 further comprising fusing the mental state metrics that were estimated to produce a smoothed mental state metric. ### Claim 18 (depends on 16) The method of claim 16 wherein the successive, overlapped speech segments are windowed around 1200 ms. ### Claim 19 (depends on 1) The method of claim 1 wherein the first group of utterances includes non-speech vocalizations. ### Claim at position 20 20-21. (canceled) ### Claim 22 (depends on 1) The method of claim 1 wherein the outputting is used for developing cross-cultural conversational agents. ### Claim 23 (depends on 22) The method of claim 22 wherein the cross-cultural conversational agents are used in vehicular control. ### Claim 24 (depends on 1) The method of claim 1 further comprising training an application for use with a third language distinct from the first language and the second language. ### Claim 25 (depends on 1) The method of claim 1 further comprising developing cross-linguistic models based on the outputting. ### Claim 26 (depends on 25) The method of claim 25 further comprising training the cross-linguistic models based on one or more human reactions to an application using the cross-linguistic models. ### Claim 27 (canceled) ### Claim 28 A computer program product embodied in a non-transitory computer readable medium for speech analysis, the computer program product comprising code which causes one or more processors to perform operations of: collecting, on a computing device, a first group of utterances in a first language with an associated first set of mental states; storing, on an electronic storage device, the first group of utterances and the associated first set of mental states; training a machine learning system using the first group of utterances and the associated first set of mental states that were stored; processing, on the machine learning system that was trained, a second group of utterances from a second language, wherein the processing determines a second set of mental states corresponding to the second group of utterances; and outputting the second set of mental states. ### Claim 29 A computer system for speech analysis comprising: a memory which stores instructions; one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to: collect, on a computing device, a first group of utterances in a first language with an associated first set of mental states; store, on an electronic storage device, the first group of utterances and the associated first set of mental states; train a machine learning system using the first group of utterances and the associated first set of mental states that were stored; process, on the machine learning system that was trained, a second group of utterances from a second language, wherein the processing determines a second set of mental states corresponding to the second group of utterances; and output the second set of mental states.