Back to results
US 12,675,641 B2Granted
Detection of interaction events in recorded audio streams
AssigneeObserve.AI, Inc.
InventorsJithendra Vepa, Krishnachaitanya Gogineni, Ayush Kumar, Digvijay Anil Ingle, Murtaza Ali, Bhasker Sharma, Sreeram Rajagopalan
Life of this patent
Estimating the term…
Abstract
Detection of interaction events in recorded audio streams is disclosed, including: detecting an interaction event within a recorded audio stream; analyzing text before and after the interaction event in the recorded audio stream to determine a causer of the interaction event; and determining an action to be performed in response to the interaction event based at least in part on the causer of the interaction event.
Claims20
A system, comprising:
- a processor configured to:
- detect an interaction event within a recorded audio stream, wherein to detect the interaction event within the recorded audio stream comprises to:
- divide the recorded audio stream into a plurality of audio frames;
- determine an audio segment based at least in part on audio frames that are included in a sliding window;
- determine that a subset of audio frames within the audio segment includes the interaction event;
- and
- determine a time segment within the recorded audio stream corresponding to a duration during which the interaction event is detected within the recorded audio stream, wherein the time segment comprises a start timestamp that denotes a start associated with the interaction event and an end timestamp that denotes an end associated with the interaction event;
- obtain a text transcription of the recorded audio stream, wherein the text transcription is time aligned with speech in the recorded audio stream and includes annotations of participants for respective speaking turns;
- analyze contextual text comprising a first portion of text in the text transcription that precedes the time segment associated with the interaction event and a second portion of text in the text transcription that follows the time segment associated with the interaction event in the recorded audio stream to determine a causer of the interaction event, wherein the first portion of text in the text transcription comprises a first predetermined number of words in the text transcription that precedes the start timestamp associated with the interaction event, and wherein the second portion of text in the text transcription comprises a second predetermined number of words in the text transcription that follows the end timestamp associated with the interaction event, wherein to determine the causer of the interaction event comprises to input the contextual text comprising the first portion of text and the second portion of text into a text-based interaction event classification model, wherein the text-based interaction event classification model is configured to output information indicating the causer of the interaction event, wherein the text-based interaction event classification model has been trained on training data comprising labeled text snippets associated with recorded interactions, wherein a labeled text snippet includes:
- contextual text surrounding a detected interaction event within the labeled text snippet, a set of first labels indicating which speaking turn in the contextual text belongs to which of a first participant role and a second participant role in a corresponding recorded interaction, a second label indicating which of the first participant role and the second participant role had caused the detected interaction event within the labeled text snippet, and a third label indicating whether the detected interaction event within the labeled text snippet was expected or unexpected;
- and
- determine an action to be performed in response to the interaction event based at least in part on the causer of the interaction event;
- and
- a memory coupled to the processor and configured to provide the processor with instructions.
11 more elements
A method, comprising:
- detecting an interaction event within a recorded audio stream, wherein detecting the interaction event within the recorded audio stream comprises:
- dividing the recorded audio stream into a plurality of audio frames;
- determining an audio segment based at least in part on audio frames that are included in a sliding window;
- determining that a subset of audio frames within the audio segment includes the interaction event;
- and
- determining a time segment within the recorded audio stream corresponding to a duration during which the interaction event is detected within the recorded audio stream, wherein the time segment comprises a start timestamp that denotes a start associated with the interaction event and an end timestamp that denotes an end associated with the interaction event;
- obtaining a text transcription of the recorded audio stream, wherein the text transcription is time aligned with speech in the recorded audio stream and includes annotations of participants for respective speaking turns;
- analyzing contextual text comprising a first portion of text in the text transcription that precedes the time segment associated with the interaction event and a second portion of text in the text transcription that follows the time segment associated with the interaction event in the recorded audio stream to determine a causer of the interaction event, wherein the first portion of text in the text transcription comprises a first predetermined number of words in the text transcription that precedes the start timestamp associated with the interaction event, and wherein the second portion of text in the text transcription comprises a second predetermined number of words in the text transcription that follows the end timestamp associated with the interaction event, wherein to determine the causer of the interaction event comprises to input the contextual text comprising the first portion of text and the second portion of text into a text-based interaction event classification model, wherein the text-based interaction event classification model is configured to output information indicating the causer of the interaction event, wherein the text-based interaction event classification model has been trained on training data comprising labeled text snippets associated with recorded interactions, wherein a labeled text snippet includes:
- contextual text surrounding a detected interaction event within the labeled text snippet, a set of first labels indicating which speaking turn in the contextual text belongs to which of a first participant role and a second participant role in a corresponding recorded interaction, a second label indicating which of the first participant role and the second participant role had caused the detected interaction event within the labeled text snippet, and a third label indicating whether the detected interaction event within the labeled text snippet was expected or unexpected;
- and
- determining an action to be performed in response to the interaction event based at least in part on the causer of the interaction event.
8 more elements
A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- detecting an interaction event within a recorded audio stream, wherein detecting the interaction event within the recorded audio stream comprises:
- dividing the recorded audio stream into a plurality of audio frames;
- determining an audio segment based at least in part on audio frames that are included in a sliding window;
- determining that a subset of audio frames within the audio segment includes the interaction event;
- and
- determining a time segment within the recorded audio stream corresponding to a duration during which the interaction event is detected within the recorded audio stream, wherein the time segment comprises a start timestamp that denotes a start associated with the interaction event and an end timestamp that denotes an end associated with the interaction event;
- obtaining a text transcription of the recorded audio stream, wherein the text transcription is time aligned with speech in the recorded audio stream and includes annotations of participants for respective speaking turns;
- analyzing contextual text comprising a first portion of text in the text transcription that precedes the time segment associated with the interaction event and a second portion of text in the text transcription that follows the time segment associated with the interaction event in the recorded audio stream to determine a causer of the interaction event, wherein the first portion of text in the text transcription comprises a first predetermined number of words in the text transcription that precedes the start timestamp associated with the interaction event, and wherein the second portion of text in the text transcription comprises a second predetermined number of words in the text transcription that follows the end timestamp associated with the interaction event, wherein to determine the causer of the interaction event comprises to input the contextual text comprising the first portion of text and the second portion of text into a text-based interaction event classification model, wherein the text-based interaction event classification model is configured to output information indicating the causer of the interaction event, wherein the text-based interaction event classification model has been trained on training data comprising labeled text snippets associated with recorded interactions, wherein a labeled text snippet includes:
- contextual text surrounding a detected interaction event within the labeled text snippet, a set of first labels indicating which speaking turn in the contextual text belongs to which of a first participant role and a second participant role in a corresponding recorded interaction, a second label indicating which of the first participant role and the second participant role had caused the detected interaction event within the labeled text snippet, and a third label indicating whether the detected interaction event within the labeled text snippet was expected or unexpected;
- and
- determining an action to be performed in response to the interaction event based at least in part on the causer of the interaction event.
8 more elements
Dependencies are inferred from the claim text.