# US 2018/0144761 A1 · Generating communicative behaviors for anthropomorphic virtual agents based on user's affect > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0144761 A1 - Document ID: US20180144761A1 - Status: Published application - Kind: A1 - Application number: 15/718,170 - Filed: 2017-09-28 - Published: 2018-05-24 - Earliest claimed: 2016-11-18 - Assignees: Ipsoft Incorporated - Applicants: IPsoft Incorporated - Inventors: Reza AMINI; Ugan YASAVUR; Jorge TRAVIESO; Chetan DUBE - Web page: https://patentgrep.com/patent/US20180144761A1 ## CPC - G10L 21/10: Transforming into visible information - G06F 40/35: Discourse or dialogue representation - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/172: {Classification, e.g. identification} - G06V 40/174: {Facial expression recognition} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 25/63: for estimating an emotional state - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent based on emotion, mood and/or personality of a user and/or the virtual agent are provided. Systems and method for determining a user's emotion, mood and/or personality are also provided. ## Claims ### Claim 1 A method for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent, the method comprising: receiving, by a computer, an emotion vector for a user, a mood vector for the user, and a personality vector of the user, wherein the emotion vector for the user, the mood vector for the user and the personality vector of the user are all for a particular utterance of the user; determining, by a computer, an emotion vector for the virtual agent based on the emotion vector of the user; determining, by the computer, a mood vector for the virtual agent based on i) the emotion vector for the virtual agent, and ii) the mood vector of the user; determining, by the computer, a personality vector for the virtual agent based on i) the mood vector for the virtual agent, and ii) a personality vector of the user; determining, by the computer, at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent; and applying, by the computer, the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both. ### Claim 2 (depends on 1) The method of claim 1 wherein the emotion vector, mood vector, and the personality vector are expressed as a pleasure, arousal, and dominance model. ### Claim 3 (depends on 1) The method of claim 1 wherein applying the facial expression, body gesture, vocal expression, verbal expression or any combination thereof to the virtual agent further comprise outputting by the virtual agent a virtual agent utterance. ### Claim 4 (depends on 1) The method of claim 1 wherein applying the facial expression, body gesture, vocal expression, verbal expressions, or any combination thereof further comprises generating behavior markup language. ### Claim 5 (depends on 1) The method of claim 1 further comprising determining, by the computer, one or more utterances for the virtual agent based on the content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent. ### Claim 6 (depends on 5) The method of claim 5 wherein determining one or more utterances is further based on a profile of the user. ### Claim 7 (depends on 5) The method of claim 5 wherein determining one or more utterances further comprises for each of the one or more utterances, determining each word of the current utterance based on previously recorded interactions. ### Claim 8 (depends on 5) The method of claim 5 wherein determining one or more utterances further comprises for each of the one or more utterances, selecting a single utterance from a group of utterance candidates based on a prediction of the next utterance. ### Claim 9 (depends on 1) The method of claim 1 wherein determining an emotion vector for the virtual agent is further based on an emotion vector of the virtual agent for a current utterance of the virtual agent, and an emotion vector for the virtual agent's last utterance. ### Claim 10 (depends on 9) The method of claim 9 wherein determining the emotion vector for the virtual agent further comprises assigning, by the computer, a weight to at least one of i) the emotion vector of the user, ii) the emotion vector of the virtual agent for a current utterance of the virtual agent, and iii) the emotion vector for the virtual agent's last utterance, based on a type of the virtual agent, an administrative user input, or any combination thereof. ### Claim 11 (depends on 1) The method of claim 1 wherein determining a mood vector for the virtual agent is further based a mood vector for the virtual agent's last utterance. ### Claim 12 (depends on 11) The method of claim 11 wherein determining a mood vector for the virtual agent further comprises assigning a weight to at least one of i) the emotion vector for the virtual agent, ii) the mood vector of the user, and iii) the mood vector for the virtual agent's last utterance, based on a type of the virtual agent, an administrative user input, or combination thereof. ### Claim 13 (depends on 1) The method of claim 1 wherein determining a personality vector for the virtual agent is further based on a personality vector for the virtual agent's last utterance. ### Claim 14 (depends on 13) The method of claim 13 wherein determining a personality vector for the virtual agent further comprises assigning, by the computer, a weight to at least one of i) mood vector for the virtual agent, ii) the personality vector of the user, and iii) the personality vector for the virtual agent's last utterance based on a type of the virtual agent, an administrative user input, or combination thereof. ### Claim 15 (depends on 1) The method of claim 1 wherein determining a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent is further based on content of an utterance of the virtual agent. ### Claim 16 An article comprising a non-transitory computer readable storage medium containing program instructions for causing a computer to perform a method for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent, the method comprising: receive an emotion vector for a user, a mood vector for the user, and a personality vector of the user, wherein the emotion vector for the user, the mood vector for the user and the personality vector of the user are all for a particular utterance of the user; determine an emotion vector for the virtual agent based on the emotion vector of the user; determine a mood vector for the virtual agent based on i) the emotion vector for the virtual agent, and ii) the mood vector of the user; determine a personality vector for the virtual agent based on i) the mood vector for the virtual agent, and ii) a personality vector of the user; determine at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent; and apply the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both. ### Claim 17 An apparatus comprising: a processor configured to: receive an emotion vector for a user, a mood vector for the user, and a personality vector of the user, wherein the emotion vector for the user, the mood vector for the user and the personality vector of the user are all for a particular utterance of the user; determine an emotion vector for the virtual agent based on the emotion vector of the user; determine a mood vector for the virtual agent based on i) the emotion vector for the virtual agent, and ii) the mood vector of the user; determine a personality vector for the virtual agent based on i) the mood vector for the virtual agent, and ii) a personality vector of the user; determine at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent; and apply the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both.