# US 10,120,866 B1 · Conversational system user behavior identification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,120,866 B1 - Document ID: US10120866B1 - Status: Granted - Kind: B1 - Application number: 15/581,969 - Filed: 2017-04-28 - Published: 2018-11-01 - Granted: 2018-11-06 - Earliest claimed: 2017-04-28 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Joseph Edwin Johnson, Jr.; Emmanouil Koukoumidis; Donald Brinkman; Matthew Schuerman - Web page: https://patentgrep.com/patent/US10120866B1 ## CPC - G06F 40/35: Discourse or dialogue representation - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06Q 10/1053: Employment or hiring - G10L 25/63: for estimating an emotional state - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Examples of the present disclosure describe systems and methods relating to conversational system user behavior identification. A user of the conversational system may be evaluated based on one or more factors. The one or more factors may be compared to an aggregated measure for a larger group of conversational system users, such that “anomalous” behavior (e.g., behavior that deviates from a normal behavior) may be identified. When a user is identified as exhibiting anomalous behavior, the conversational system may adapt its interactions with the user in order to encourage, discourage, or further observe the identified behavior. As a result, the conversational system may be able to verify a user's anomalous behavior, discourage the anomalous behavior, or take other action while interacting with the user. ## Claims ### Claim 1 A system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising: receiving, as part of a conversation session, a message from a user device, wherein the conversation session is between a user of the user device and an electronic conversational agent; determining, based on the message and one or more factors associated with the conversation session, a behavior measure associated with the user; determining, based on analyzing the behavior measure in relation to one or more historical factors associated with a subset of other users of the electronic conversational agent, whether the user is exhibiting anomalous behavior; when it is determined that the user is exhibiting anomalous behavior, automatically adapting the electronic conversational agent based on the determined anomalous behavior; and continuing the conversation session based on the adapted electronic conversational agent. ### Claim 2 (depends on 1) The system of claim 1, wherein determining the behavior measure comprises: computing one or more averages for the one or more factors; and generating the behavior measure based on an evaluation of the one or more averages using one or more thresholds. ### Claim 3 (depends on 1) The system of claim 1, wherein automatically adapting the electronic conversational agent based on the determined anomalous behavior comprises: identifying, from one or more potential responses to the message, a response associated with the determined anomalous behavior; and evaluating a reaction of the user to the response to determine whether the user continues to exhibit the determined anomalous behavior. ### Claim 4 (depends on 1) The system of claim 1, wherein automatically adapting the electronic conversational agent based on the determined anomalous behavior comprises: providing a challenge to the user device, wherein the challenge comprises at least a first part and a second part, wherein the first part is associated with an expected response; receiving, from the user device, a challenge response, wherein the challenge response comprises a first response part associated with the first part and a second response part associated with the second part; and determining, based on comparing the first response part to the expected response, a validity for the second response part. ### Claim 5 (depends on 1) The system of claim 1, wherein the set of operations further comprises: receiving a second message from the user device; determining, based on the second message and one or more factors associated with the conversation session, a second behavior measure associated with the user; determining, based on the second behavior measure, whether the user continues to exhibit anomalous behavior; and when it is determined that the user does not exhibit anomalous behavior, adapting the electronic conversational agent to a normal state. ### Claim 6 (depends on 1) The system of claim 1, wherein the one or more factors associated with the conversation session comprise one or more factors associated with a previous conversation session between the user and the electronic conversational agent. ### Claim 7 (depends on 1) The system of claim 1, wherein the conversational session is an interview with the user. ### Claim 8 A computer-implemented method for behavior evaluation of a user in a conversation session, comprising: receiving, by a processor of a computer, a message from a user device, wherein the conversation session is between the user and an electronic conversational agent; updating, based on the conversation session and one or more factors associated with the message, a behavior measure associated with the user; determining, based on analyzing the behavior measure in relation to one or more historical factors associated with a subset of other users of the electronic conversational agent, whether the user is exhibiting anomalous behavior; and when it is determined that the user is exhibiting anomalous behavior, providing an indication to the electronic conversational agent to adapt one or more future communications with the user device based on the determined anomalous behavior. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein updating the behavior measure comprises: computing one or more averages for the one or more factors; and generating the behavior measure based on an evaluation of the one or more averages using one or more thresholds. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein the indication to the electronic conversational agent comprises an indication to: identify, from one or more potential responses to the message, a response associated with the determined anomalous behavior; and evaluate a reaction of the user to the response to determine whether the user continues to exhibit the determined anomalous behavior. ### Claim 11 (depends on 8) The computer-implemented method of claim 8, wherein the indication to the electronic conversational agent comprises an indication to: provide a challenge for the user to the user device, wherein the challenge comprises at least a first part and a second part, wherein the first part is associated with an expected response; receive, from the user device, a challenge response, wherein the challenge response comprises a first response part associated with the first part and a second response part associated with the second part; and determine, based on comparing the first response part to the expected response, a validity for the second response part. ### Claim 12 (depends on 8) The computer-implemented method of claim 8, further comprising: receiving a second message from the user device; updating, based on the conversation session and one or more factors associated with the second message, the behavior measure associated with the user; determining, based on the behavior measure, whether the user continues to exhibit anomalous behavior; and when it is determined that the user does not exhibit anomalous behavior, providing an indication to the electronic conversational agent to enter a normal state. ### Claim 13 (depends on 8) The computer-implemented method of claim 8, wherein the one or more factors associated with the message comprise one or more factors associated with a previous message from at least one of the conversation session and a previous conversation session between the user and the conversational agent. ### Claim 14 A computer-implemented method for behavior evaluation of a user in a conversation session, comprising: receiving, by a processor of a computer as part of the conversation session, a message from a user device, wherein the conversation session is between the user and an electronic conversational agent; determining, based on the message and one or more factors associated with the conversation session, a behavior measure associated with the user; determining, based on analyzing the behavior measure in relation to one or more historical factors associated with a subset of other users of the electronic conversational agent, whether the user is exhibiting anomalous behavior; when it is determined that the user is exhibiting anomalous behavior, automatically adapting the electronic conversational agent based on the determined anomalous behavior; and continuing the conversation session based on the adapted electronic conversational agent. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein determining the behavior measure comprises: computing one or more averages for the one or more factors; and generating the behavior measure based on an evaluation of the one or more averages using one or more thresholds. ### Claim 16 (depends on 14) The computer-implemented method of claim 14, wherein automatically adapting the electronic conversational agent based on the determined anomalous behavior comprises: identifying, from one or more potential responses to the message, a response associated with the determined anomalous behavior; and evaluating a reaction of the user to the response to determine whether the user continues to exhibit the determined anomalous behavior. ### Claim 17 (depends on 14) The computer-implemented method of claim 14, wherein automatically adapting the electronic conversational agent based on the determined anomalous behavior comprises: providing a challenge to the user device, wherein the challenge comprises at least a first part and a second part, wherein the first part is associated with an expected response; receiving, from the user device, a challenge response, wherein the challenge response comprises a first response part associated with the first part and a second response part associated with the second part; and determining, based on comparing the first response part to the expected response, a validity for the second response part. ### Claim 18 (depends on 14) The computer-implemented method of claim 14, further comprising: receiving a second message from the user device; determining, based on the second message and one or more factors associated with the conversation session, a second behavior measure associated with the user; determining, based on the second behavior measure, whether the user continues to exhibit anomalous behavior; and when it is determined that the user does not exhibit anomalous behavior, adapting the electronic conversational agent to a normal state. ### Claim 19 (depends on 14) The computer-implemented method of claim 14, wherein the one or more factors associated with the conversation session comprise one or more factors associated with a previous conversational session between the user and the electronic conversational agent. ### Claim 20 (depends on 15) The computer-implemented method of claim 15, wherein the message was received in response to a prompt by the electronic conversational agent, and wherein the one or more thresholds are determined based on a confidence level associated with at least one of the message and the prompt.