Multi-modal modeling of temporal interaction sequences
AssigneeSRI International
InventorsAjay Divakaran, Behjat Siddiquie, Saad Khan, Jeffrey Lubin, Harpreet S. Sawhney
Life of this patent
Estimating the term…
Abstract
A multi-modal interaction modeling system can model a number of different aspects of a human interaction across one or more temporal interaction sequences. Some versions of the system can generate assessments of the nature or quality of the interaction or portions thereof, which can be used to, among other things, provide assistance to one or more of the participants in the interaction.
Claims18
A method for predicting a behavioral event to occur in an interaction involving at least two participants, at least one of the participants being a person, the method comprising, with a computing system:
- detecting, from multi-modal data captured by one or more sensing devices, a plurality of different behavioral cues expressed by the participants during the interaction;
- recognizing a plurality of temporal interaction sequences, each of the temporal interaction sequences occurring over a time interval during the interaction, and involving a pattern of the behavioral cues, and at least two of the participants being involved in at least one of the temporal interaction sequences;
- determining the nature of the interaction based on the recognized temporal interaction sequences;
- and
- predicting the behavioral event to occur in the interaction based on the determined nature of the interaction.
An interaction assistant embodied in one or more machine-readable storage media and accessible by a computing device to assist with an interaction involving a natural-language dialog between a person and the computing device, by:
- detecting, from multi-modal data captured by at least one sensing device, a plurality of different behavioral cues expressed by the person during the natural-language dialog;
- recognizing a temporal interaction sequence comprising a pattern of the behavioral cues occurring over a time interval during the natural-language dialog;
- deriving, from the temporal interaction sequence, an assessment of a portion of the natural-language dialog involving the person;
- and
- formulating a portion of the natural-language dialog involving the computing device based on the assessment.
A method for assessing a person's emotional state during an interaction involving the person and at least one other participant, the method comprising, with a computing system:
- detecting, from multi-modal data captured by one or more sensing devices, a plurality of different behavioral cues expressed by the participants during the interaction;
- recognizing a plurality of temporal interaction sequences, each temporal interaction sequence comprising a pattern of the behavioral cues occurring over a time interval during the interaction, and at least one of the temporal interaction sequences involving the person and at least one other participant;
- assessing the person's emotional state during each of the temporal interaction sequences based on the behavioral cues involved in the temporal interaction sequence;
- detecting changes in the participant's behavior over a period of time defining the duration of the interaction;
- and
- evaluating the participant's behavior over the period of time based on the detected changes in behavior and the time intervals in which they occurred.
Dependencies are inferred from the claim text.