# US 2014/0191939 A1 · Using nonverbal communication in determining actions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0191939 A1 - Document ID: US20140191939A1 - Status: Published application - Kind: A1 - Application number: 13/737,542 - Filed: 2013-01-09 - Published: 2014-07-10 - Earliest claimed: 2013-01-09 - Assignees: Microsoft Corporation - Applicants: Microsoft Corporation - Inventors: Daniel J. Penn; Mark Hanson; Robert Chambers; Elizabeth Shriberg - Web page: https://patentgrep.com/patent/US20140191939A1 ## CPC - G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Nonverbal communication is used when determining an action to perform in response to received user input. The received input includes direct input (e.g. speech, text, gestures) and indirect input (e.g. nonverbal communication). The nonverbal communication includes cues such as body language, facial expressions, breathing rate, heart rate, well as vocal cues (e.g. prosodic and acoustic cues) and the like. Different nonverbal communication cues are monitored such that performed actions are personalized. A direct input specifying an action to perform (e.g. “perform action 1”) may be adjusted based on one or more indirect inputs (e.g. nonverbal cues) received. Another action may also be performed in response to the indirect inputs. A profile may be associated with the user such that the responses provided by the system are determined using nonverbal cues that are associated with the user. ## Claims ### Claim 1 A method for using nonverbal communication to determine an intended action, comprising: receiving user interaction comprising direct input specifying an intended action and indirect input comprising nonverbal communication; determining the direct input using at least one of: speech input, gesture input, and textual input; determining the indirect input comprising the nonverbal communication; using the indirect communication in addition to the intended action determined from the direct input to determine an action to perform; and performing the action. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining a user satisfaction using received nonverbal communication after performing the action. ### Claim 3 (depends on 1) The method of claim 1, further comprising performing an additional action in response to determining a user satisfaction using received nonverbal communication after performing the action. ### Claim 4 (depends on 3) The method of claim 3, wherein performing the additional action in response to determining the user satisfaction comprises requesting a clarification to the intended action. ### Claim 5 (depends on 1) The method of claim 1, wherein performing the additional action in response to determining the user satisfaction comprises changing content displayed on a user interface. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the user satisfaction using received nonverbal communication after performing the action comprises determining a facial expression. ### Claim 7 (depends on 1) The method of claim 1, wherein the nonverbal communication comprises a voice cue comprising one or more of: a voice level, a spacing of speech, and a rate of speech. ### Claim 8 (depends on 1) The method of claim 1, further comprising accessing a profile of a user associated with the input that includes nonverbal communication information that is associated with the user. ### Claim 9 (depends on 1) The method of claim 1, wherein the nonverbal communication comprises one or more of: a vocal cue, a heart rate, a breathing rate, a facial expression, a body movement, and a posture. ### Claim 10 A computer-readable medium storing computer-executable instructions for using nonverbal communication, comprising: receiving user interaction comprising direct input specifying an intended action and indirect input comprising nonverbal communication; determining the direct input using at least one of: speech input, gesture input, and textual input; determining the indirect input comprising the nonverbal communication that comprises one or more of: a vocal cue, a heart rate, a breathing rate, a facial expression, a body movement, and a posture; accessing a profile that includes information relating to a baseline of nonverbal communication cues associated with a user; determining changes from the baseline using the determined indirect communication; using the indirect communication and determined changes in addition to the intended action determined from the direct input to determine an action to perform; and performing the action. ### Claim 11 (depends on 10) The computer-readable medium of claim 10, further comprising determining a user satisfaction using received nonverbal communication after performing the action. ### Claim 12 (depends on 10) The computer-readable medium of claim 10, further comprising performing an additional action in response to determining a user satisfaction using received nonverbal communication after performing the action. ### Claim 13 (depends on 12) The computer-readable medium of claim 12, wherein performing the additional action in response to determining the user satisfaction comprises requesting a clarification to the intended action. ### Claim 14 (depends on 10) The computer-readable medium of claim 10, wherein performing the additional action in response to determining the user satisfaction comprises changing content displayed on a user interface. ### Claim 15 (depends on 10) The computer-readable medium of claim 10, wherein determining the user satisfaction using received nonverbal communication after performing the action comprises determining a facial expression. ### Claim 16 (depends on 10) The computer-readable medium of claim 10, wherein the nonverbal communication comprises a voice cue comprising one or more of: a voice level, a spacing of speech, and a rate of speech. ### Claim 17 A system for using nonverbal communication, comprising: a camera that is configured to detect movements; a microphone that is configured to receive speech input; a processor and memory; an operating environment executing using the processor; a display; and an understanding manager that is configured to perform actions comprising: receiving user interaction comprising direct input specifying an intended action and indirect input comprising nonverbal communication; determining the direct input using at least one of: speech input, gesture input, and textual input; determining the indirect input comprising the nonverbal communication that comprises one or more of: a vocal cue, a heart rate, a breathing rate, a facial expression, a body movement, and a posture; accessing a profile that includes information relating to a baseline of nonverbal communication cues associated with a user; determining changes from the baseline using the determined indirect communication; using the indirect communication and determined changes in addition to the intended action determined from the direct input to determine an action to perform; and performing the action. ### Claim 18 (depends on 17) The system of claim 17, further comprising determining a user satisfaction using received nonverbal communication after performing the action and performing an additional action in response to determining a user satisfaction using received nonverbal communication after performing the action. ### Claim 19 (depends on 17) The system of claim 17, wherein determining the user satisfaction using received nonverbal communication after performing the action comprises determining a facial expression. ### Claim 20 (depends on 17) The system of claim 17, wherein the nonverbal communication comprises a voice cue comprising one or more of: a voice level, a spacing of speech, and a rate of speech.