# US 12,217,758 B2 · Automated systems and methods that generate affect-annotated timelines > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,758 B2 - Document ID: US12217758B2 - Status: Granted - Kind: B2 - Application number: 18/420,383 - Filed: 2024-01-23 - Published: 2024-05-16 - Granted: 2025-02-04 - Earliest claimed: 2020-08-25 - Assignees: Affective Software, Inc. - Applicants: Affective Software, Inc. - Inventors: Vladimir Brayman; John Gottman; Connor Eaton; Yuriy Gulak; Rafael Lisitsa - Web page: https://patentgrep.com/patent/US12217758B2 ## CPC - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 25/63: for estimating an emotional state - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/04: Segmentation; Word boundary detection - G10L 25/30: using neural networks - G10L 2015/221: {Announcement of recognition results} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The current document is directed to a methods and systems that use observational data collected by various devices and sensors to generate electronic-data representations of human conversations. The implementations of these methods and systems, disclosed in the current document, provide a highly extensible and generic platform for converting observational data into affect-annotated-timeline outputs that provide both a textual transcription of a conversation and a parallel set of affect annotations to the conversation. The affect-annotated-timeline outputs may be useful to researchers and developers, but also serve as inputs to any of a wide variety of downstream analytical processes and analysis systems that are, in turn, incorporated into many different types of special-purpose analysis and control systems. ## Claims ### Claim 1 An affect-annotated-timeline data structure stored in one of a data-storage device, a data-storage appliance, and an electronic memory within a computer system, the affect-annotated-timeline data structure generated from a monitored conversation by an automated affect-annotation system, the affect-annotated-timeline data structure comprising: multiple affect-annotation records, each affect-annotation record representing a conversation unit that corresponds to a subinterval of the monitored conversation, that is attributed to a particular source, and that is generated, by the automated affect-annotation system, from one or more units of language for affect coding, referred to as “ULACs,” each ULAC corresponding to a minimal aggregation of words extracted from a conversation that conveys an intra-contextual meaning, and identified by the automated affect-annotation system based on lexical dependency graphs, and containing a textual representation of the conversation unit, an indication of a source of the conversation unit, an indication of a temporal duration of the conversation unit and an indication of a temporal position of the conversation unit within the monitored conversation, and an affect-code probability distribution. ### Claim 2 (depends on 1) The affect-annotated-timeline data structure of claim 1 wherein a source of a conversation unit is one of: a human conversation participant; and an automated-system conversation participant. ### Claim 3 (depends on 2) The affect-annotated-timeline data structure of claim 2 wherein the monitored conversation includes two or more sources. ### Claim 4 (depends on 1) The affect-annotated-timeline data structure of claim 1 wherein an indication of a temporal duration of the conversation unit and an indication of a temporal position of the conversation unit within the monitored conversation includes one of: an indication of a start time and an indication of a time-interval length; an indication of an end time and an indication of a time-interval length; and an indication of a start time and an indication of an end time. ### Claim 5 (depends on 1) The affect-annotated-timeline data structure of claim 1 wherein an affect-code probability distribution is a discrete probability distribution that indicates the probability that the conversation unit represented by an affect-annotation record corresponds to each of multiple affect codes that each corresponds to an emotional state and/or intention that corresponds to one or more observables, the observables including behavioral patterns and physiological conditions. ### Claim 6 (depends on 5) The affect-annotated-timeline data structure of claim 5 wherein the affect codes are each represented by a numerical indication unique to the affect code. ### Claim 7 (depends on 6) The affect-annotated-timeline data structure of claim 6 wherein the affect codes include: negative affect codes that include contempt/belligerence, domineering/defensive/anger, sadness/whining, and tension; a neutral affect code; and positive affect codes that include validation, interest, joy/affection, and humor. ### Claim 8 (depends on 1) The affect-annotated-timeline data structure of claim 1 wherein each affect-annotation record further includes a sequence number. ### Claim 9 (depends on 8) The affect-annotated-timeline data structure of claim 8 wherein each affect-annotation record further includes one or more additional numeric, text-containing, and alphanumeric fields. ### Claim 10 An affect-annotation system that receives conversation data generated from a monitored conversation and processes the input conversation data to produce an electronic representation of the monitored conversation, the affect-annotation system comprising: one or more processors; one or more memories and mass-storage devices; and computer instructions, stored in one or more of the one or more memories and mass-storage devices that, when executed by one or more of the one or more processors, control the affect-annotation system to receive conversation data, including one or more of video data, audio data, and physiological data, generate multiple affect-annotation records for each of multiple conversation units identified using the received conversation data, each conversation unit corresponding to a subinterval of the monitored conversation, attributed to a particular source, generated, by the automated affect-annotation system, from one or more units of language for affect coding, referred to as “ULACs,” each ULAC corresponding to a minimal aggregation of words extracted from a conversation that conveys an intra-contextual meaning, and identified by the automated affect-annotation system based on lexical dependency graphs, and each affect-annotation record containing a textual representation of a conversation unit, an indication of a source of the conversation unit, an indication of a temporal duration of the conversation unit, an indication of a temporal position of the conversation unit within the monitored conversation, and an affect-code probability distribution, aggregate the generated affect-annotation records into an affect-annotated-timeline data structure, and output the affect-annotated-timeline data structure to one or more of a display device, a data-storage appliance or device, and one or more downstream analysis systems. ### Claim 11 (depends on 10) The affect-annotation system of claim 10 wherein a source of a conversation unit is one of a human conversation participant; and an automated-system conversation participant; and wherein the monitored conversation includes two or more sources. ### Claim 12 (depends on 10) The affect-annotation system of claim 10 wherein an indication of a temporal duration of the conversation unit and an indication of a temporal position of the conversation unit within the monitored conversation includes one of: an indication of a start time and an indication of a time-interval length; an indication of an end time and an indication of a time-interval length; and an indication of a start time and an indication of an end time. ### Claim 13 (depends on 10) The affect-annotation system of claim 10 wherein an affect-code probability distribution is a discrete probability distribution that indicates the probability that the conversation unit represented by an affect-annotation record corresponds to each of multiple affect codes that each corresponds to an emotional state and/or intention that corresponds to one or more observables, the observables including behavioral patterns and physiological conditions. ### Claim 14 (depends on 10) The affect-annotation system of claim 10 wherein the affect codes are each represented by a numerical indication unique to the affect code. ### Claim 15 (depends on 14) The affect-annotation system of claim 14 wherein the affect codes include: negative affect codes that include contempt/belligerence, domineering/defensive/anger, sadness/whining, and tension; a neutral affect code; and positive affect codes that include validation, interest, joy/affection, and humor. ### Claim 16 (depends on 10) The affect-annotation system of claim 10 wherein each affect-annotation record further includes a sequence number.