# US 9,437,215 B2 · Predictive video analytics system and methods > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,437,215 B2 - Document ID: US9437215B2 - Status: Granted - Kind: B2 - Application number: 14/996,913 - Filed: 2016-01-15 - Published: 2016-05-12 - Granted: 2016-09-06 - Earliest claimed: 2014-10-27 - Assignees: Mattersight Corporation - Applicants: Mattersight Corporation - Inventors: Kelly Conway; Christopher Danson - Web page: https://patentgrep.com/patent/US9437215B2 ## CPC - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - H04M 3/51: Centralised call answering arrangements requiring operator intervention {, e.g. call or contact centers for telemarketing} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 40/174: {Facial expression recognition} - G06V 40/18: Eye characteristics, e.g. of the iris - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 17/22: Interactive procedures; Man-machine interfaces - G10L 17/26: Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices - G10L 25/57: for processing of video signals - G10L 25/63: for estimating an emotional state - H04M 3/2281: {Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls} - G06T 2207/10016: Video; Image sequence - G06T 2207/30201: Face - G06V 40/179: {metadata assisted face recognition} - H04M 2203/357: Autocues for dialog assistance - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract The methods and systems described herein predict user behavior based on analysis of a user video communication. The methods include receiving a user video communication, extracting video facial analysis data from the video communication, extracting voice analysis data from the video communication, associating the video facial analysis data with the voice analysis data to determine an emotional state of a user, collecting biographical profile information specific to the user, applying a linguistic-based psychological behavioral model to the spoken words to determine personality type of the user, and inputting the collected biographical profile information, emotional state, and personality type into a predictive model to determine a likelihood of an outcome of the video communication. ## Claims ### Claim 1 A video analytics system adapted to predict user behavior based on analysis of a video communication, which comprises: a node comprising a processor and a non-transitory computer readable medium operably coupled thereto, the non-transitory computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, where the plurality of instructions comprises: instructions that, when executed, receive a video communication from a user, wherein the video communication comprises an audio component and a video component; instructions that, when executed, analyze the video component to provide time-coded video behavioral data from the user; instructions that, when executed, analyze the audio component to provide time-coded spoken words from the user; instructions that, when executed, associate the time-coded spoken words with the video behavioral data to determine an emotional state of the user; instructions that, when executed, collect biographical profile information specific to the user; instructions that, when executed, determine a personality type of the user by applying a linguistic-based algorithm to the spoken words, searching a density of keywords in the spoken words, and comparing the keywords to a library separated by different personality types; and instructions that, when executed, enter the collected biographical profile information, the emotional state, and the personality type into a predictive model, wherein the predictive model generates an indication of a likelihood of an outcome of the video communication. ### Claim 2 (depends on 1) The system of claim 1, wherein the outcome comprises one or more of whether a user will terminate his or her account, whether a user will purchase a product, whether a user is a fraudster, and whether a user will initiate additional subsequent interaction sessions regarding an issue. ### Claim 3 (depends on 1) The system of claim 1, wherein the video component includes a non-verbal, non-textual element comprising one or more of eye movement, facial expressions, gestures, activities, body postures, behaviors, attire, and actions. ### Claim 4 (depends on 1) The system of claim 1, wherein more than one user is speaking in the video communication. ### Claim 5 (depends on 4) The system of claim 4, wherein the audio and video components are analyzed simultaneously to determine which user is speaking. ### Claim 6 (depends on 1) The system of claim 1, which further comprises instructions, that when executed, identify attire of the user. ### Claim 7 (depends on 6) The system of claim 6, which further comprises instructions, that when executed, determine user value data based on estimated cost of the attire. ### Claim 8 (depends on 6) The system of claim 6, which further comprises instructions, that when executed, recommend items based on the attire of the user and display the items to an agent. ### Claim 9 (depends on 1) The system of claim 1, which further comprises instructions, that when executed, determine distress and engagement of the user. ### Claim 10 (depends on 9) The system of claim 9, wherein the distress and engagement are entered into the predictive model. ### Claim 11 (depends on 1) The system of claim 1, which further comprises instructions, that when executed, build the predictive model using previously-obtained emotional states and personality types of a plurality of different users. ### Claim 12 (depends on 11) The system of claim 11, wherein the emotional states are aggregated by personality types. ### Claim 13 (depends on 1) The system of claim 1, which further comprises instructions, that when executed: provide the indication of the outcome to an agent; determine, based at least in part on the indication of the likelihood of the outcome of the video communication, whether the agent should speak specific words, perform one or more specific actions, or both; and provide the agent with the specific words, the one or more specific actions, or both. ### Claim 14 (depends on 1) The system of claim 1, which further comprises instructions, that when executed: determine the likelihood of the outcome of the video communication should be increased or decreased; select a second agent for transferring the video communication to, wherein the transferring of the video communication to the second agent is determined, using at least the predictive model, to increase or decrease the likelihood of the outcome occurring; and transfer the video communication to the second agent. ### Claim 15 A method to predict user behavior based on analysis of a video communication, which comprises: receiving, by one or more processors, a user video communication; extracting, by the one or more processors, video facial analysis data for the user from the video communication; extracting, by the one or more processors, voice analysis data including spoken words from the user video communication; associating, by the one or more processors, the video facial analysis data with the voice analysis data to determine an emotional state of the user; collecting, by the one or more processors, biographical profile information specific to the user; determining, by the one or more processors, a personality type of the user by applying a linguistic-based algorithm to the spoken words, searching a density of keywords in the spoken words, and comparing the keywords to a library separated by different personality types; and entering, by the one or more processors, the collected biographical profile information, the emotional state, and the personality type into a predictive model, wherein the predictive model generates an indication of a likelihood of an outcome of the video communication. ### Claim 16 (depends on 15) The method of claim 15, wherein the outcome comprises one or more of whether a user will terminate his or her account, whether a user will pay an outstanding bill, whether a user is a fraudster, and whether a user will initiate additional subsequent interaction sessions regarding an issue. ### Claim 17 (depends on 15) The method of claim 15, which further comprises analyzing the user's clothing and accessories. ### Claim 18 (depends on 17) The method of claim 17, which further comprises determining user value data based on the analysis of the user's clothing and accessories. ### Claim 19 (depends on 15) The method of claim 15, which further comprises generating time-coded distress and engagement data for the user. ### Claim 20 (depends on 19) The method of claim 19, which further comprises aggregating the time-coded distress and engagement data with the video facial analysis data and the voice analysis data. ### Claim 21 (depends on 15) The method of claim 15, which further comprises displaying the outcome to an agent and providing the agent with specific words, specific actions, or both. ### Claim 22 (depends on 15) The method of claim 15, which further comprises automatically sending an email with a special offer to the user based on the outcome. ### Claim 23 (depends on 15) The method of claim 15, which further comprises generating an agent performance score or generating agent training materials based on the outcome. ### Claim 24 A non-transitory machine-readable medium comprising instructions which, in response to a computer system, cause the computer system to perform a method which comprises: receiving a user video communication; separating an audio component from a video component of the video communication; analyzing facial expressions of the user in the video component; transcribing words spoken of the user in the audio component; associating the facial expressions and spoken words to determine an emotional state of the user; collecting biographical profile information specific to the user; determining a personality type of the user by applying a linguistic-based algorithm to the spoken words, searching a density of keywords in the spoken words, and comparing the keywords to a library separated by different personality types; and entering the collected biographical profile information, the emotional state, and the personality type into a predictive model, wherein the predictive model generates an indication of a likelihood of an outcome of the video communication. ### Claim 25 (depends on 24) The non-transitory machine-readable medium of claim 24, wherein the method further comprises aggregating the facial expressions and spoken words of a plurality of different users by personality type. ### Claim 26 (depends on 25) The non-transitory machine-readable medium of claim 25, wherein the method further comprises building a predictive model based on the aggregated facial expressions, spoken words, and personality type. ### Claim 27 (depends on 24) The non-transitory machine-readable medium of claim 24, wherein the method further comprises identifying the attire of the user. ### Claim 28 (depends on 27) The non-transitory machine-readable medium of claim 27, wherein the method further comprises aggregating the identified attire with the facial expressions and spoken words from a plurality of users. ### Claim 29 (depends on 24) The non-transitory machine-readable medium of claim 24, wherein the method further comprises determining distress and engagement data of the user. ### Claim 30 (depends on 29) The non-transitory machine-readable medium of claim 29, wherein the method further comprises aggregating the distress and engagement data with the facial expressions and spoken words.