# US 2021/0343282 A1 · Detecting and assigning action items to conversation participants in real-time and detecting completion thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343282 A1
- Document ID: US20210343282A1
- Status: Published application
- Kind: A1
- Application number: 17/244,804
- Filed: 2021-04-29
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: Outreach Corporation
- Applicants: Outreach Corporation
- Inventors: Rohit Ganpat Mane; Abhishek Abhishek; Krishnamohan Reddy Nareddy; Rajiv Garg
- Web page: https://patentgrep.com/patent/US20210343282A1

## CPC

- G06Q 10/103: {Workflow collaboration or project management}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/063: {Training}
- G10L 25/51: for comparison or discrimination
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 17/00: Speaker identification or verification techniques
- H04M 3/569: {using the instant speaker's algorithm (speech detection per se G10L25/78)}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

Described herein is a system for automatically detecting and assigning action items in a real-time conversation and determining whether such action items have been completed. The system detects, during a meeting, a plurality of action items and an utterance that corresponds to a completed action item. Responsive to detecting the utterance, the system generates a similarity score with respect to a first action item of the plurality of action items. The system compares the similarity score to a first threshold. Responsive to determining that the similarity score does not exceed the first threshold, the system generates a second similarity score with respect to a second action item of the plurality of action items. The system compares the second similarity score to a second threshold, which exceeds the first threshold. Responsive to determining that the second similarity score exceeds the second threshold, the system marks the second action item as completed.

## Claims

### Claim 1

A non-transitory computer readable storage medium comprising stored instructions, the instructions when executed by one or more processors cause the one or more processors to:
detect, during a meeting, a plurality of action items;
detect, during the meeting, an utterance that corresponds to a completed action item;
generate, in response to a determination of the utterance, first a similarity score with respect to a first action item of the plurality of action items;
compare the first similarity score to a first threshold;
generate, in response a determination that the first similarity score does not exceed the first threshold, a second similarity score with respect to a second action item of the plurality of action items;
compare the second similarity score to a second threshold, the second threshold exceeding the first threshold; and
mark, in response to a determination that the second similarity score exceeds the second threshold, the second action item completed.

### Claim 2 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the first action item is a most recently uttered action item, wherein the second action item is a next most recently uttered action item with respect to the most recently uttered action item, and wherein the second threshold is weighted based on an amount of time that has passed since the next most recently uttered action item was uttered.

### Claim 3 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the instructions to detect, during a meeting, the plurality of action items, further cause the one or more processors to detect each of the plurality of action items within a threshold amount of time from a current time.

### Claim 4 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:
receive, from an administrator, a threshold number and threshold amount of time;
determine a subset of the plurality of action items, wherein the subset includes fewer than the threshold number of action items and each action item of the subset occurred within a threshold amount of time from a time that the utterance was detected; and
generate similarity scores for action items in the subset based on recency of the action items.

### Claim 5 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the instructions to generate the first similarity score further cause the one or more processors to generate the first similarity score based on cosine similarity between vectors that represent the first action item and the completed action item.

### Claim 6 (depends on 5)

The non-transitory computer readable storage medium of claim 5, wherein the vectors are created using word embeddings.

### Claim 7 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:
transmit a notification, via a user interface at a client device, to an owner of the second action item that the action item has been completed.

### Claim 8 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the first similarity score is generated using a machine learning model, the machine learning model trained on a plurality of pairings of action items and utterances, wherein each pairing is labeled with a similarity score.

### Claim 9 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the first similarity scores is generated using fuzzy logic.

### Claim 10 (depends on 1)

The non-transitory computer readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:
rank the first action item higher in a ranking of the plurality of action items, the ranking determined based on a recency of when the action items were detected.

### Claim 11

A method for closing completed action items during a meeting, the method comprising:
detecting, during a meeting, a plurality of action items;
detecting, during the meeting, an utterance that corresponds to a completed action item;
generate, in response to detection of the utterance, a first similarity score with respect to a first action item of the plurality of action items;
comparing the first similarity score to a first threshold;
generate, in response to a determination that the first similarity score does not exceed the first threshold, a second similarity score with respect to a second action item of the plurality of action items;
comparing the second similarity score to a second threshold, the second threshold exceeding the first threshold; and
mark, in response to a determination that the second similarity score exceeds the second threshold, the second action item completed.

### Claim 12 (depends on 11)

The method of claim 11, wherein the first action item is a most recently uttered action item, wherein the second action item is a next most recently uttered action item with respect to the most recently uttered action item, and wherein the second threshold is weighted based on an amount of time that has passed since the next most recently uttered action item was uttered.

### Claim 13 (depends on 11)

The method of claim 11, wherein detecting, during a meeting, each of the plurality of action items further comprises detecting each of the plurality of action items within a threshold amount of time from a current time.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
receiving, from an administrator, a threshold number and threshold amount of time;
determining a subset of the plurality of action items, wherein the subset includes fewer than the threshold number of action items and each action item of the subset occurred within a threshold amount of time from a time that the utterance was detected; and
generating similarity scores for action items in the subset based on recency of the action items.

### Claim 15 (depends on 11)

The method of claim 11, wherein generating the first similarity score further comprises generating the first similarity score based on cosine similarity between vectors representing the first action item and the completed action item.

### Claim 16 (depends on 15)

The method of claim 15, wherein the vectors are created using word embeddings.

### Claim 17 (depends on 11)

The method of claim 11, further comprising:
transmitting a notification, via a user interface at a client device, to an owner of the second action item that the action item has been completed.

### Claim 18 (depends on 11)

The method of claim 11, wherein the first similarity score is generated using a machine learning model, the machine learning model trained on a plurality of pairings of action items and utterances, wherein each pairing is labeled with a similarity score.

### Claim 19 (depends on 11)

The method of claim 11, wherein the first similarity score is generated using fuzzy logic.

### Claim 20

A system for closing completed action items during a meeting, the system comprising:
a first detection module for detecting, during a meeting, a plurality of action items;
a second detection module for detecting, during the meeting, an utterance that corresponds to a completed action item;
a similarity score generation module for:
generating, in response to a detection of the utterance, a first similarity score with respect to a first action item of the plurality of action items;
comparing the first similarity score to a first threshold;
generating, in response to a determination that the first similarity score does not exceed the first threshold, a second similarity score with respect to a second action item of the plurality of action items; and
comparing the second similarity score to a second threshold, the second threshold exceeding the first threshold; and
an action item completion module for marking, in response to a determination that the second similarity score exceeds the second threshold, the second action item completed.