# US 2021/0264900 A1 · Computationally reacting to a multiparty conversation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0264900 A1
- Document ID: US20210264900A1
- Status: Published application
- Kind: A1
- Application number: 16/798,248
- Filed: 2020-02-21
- Published: 2021-08-26
- Earliest claimed: 2020-02-21
- Assignees: BetterUp, Inc.
- Applicants: BetterUp, Inc.
- Inventors: Andrew Reece; Peter Bull; Gus Cooney; Casey Fitzpatrick; Gabriella Rosen Kellerman; Ryan Sonnek
- Web page: https://patentgrep.com/patent/US20210264900A1

## CPC

- G06F 40/35: Discourse or dialogue representation
- G10L 15/063: {Training}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/041: {Abduction}
- G06N 20/00: Machine learning
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 30/268: {Lexical context}
- G09B 5/12: different stations being capable of presenting different information simultaneously (G09B5/14 takes precedence)
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15)
- H04L 65/80: Responding to QoS
- G06N 3/045: Combinations of networks
- G06Q 10/06398: {Performance of employee with respect to a job function}
- G06V 40/113: {Recognition of static hand signs}
- G06V 40/174: {Facial expression recognition}
- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Technology is provided for causing a computing system to extract conversation features from a multiparty conversation (e.g., between a coach and mentee), apply the conversation features to a machine learning system to generate conversation analysis indicators, and apply a mapping of conversation analysis indicators to actions and inferences to determine actions to take or inferences to make for the multiparty conversation. In various implementations, the actions and inferences can include determining scores for the multiparty conversation such as a score for progress toward a coaching goal, instant scores for various points throughout the conversation, conversation impact score, ownership scores, etc. These scores can be, e.g., surfaced in various user interfaces along with context and benchmark indicators, used to select resources for the coach or mentee, used to update coach/mentee matchings, used to provide real-time alerts to signify how the conversation is going, etc.

## Claims

### Claim 1

A method for computationally reacting to conversations, the method comprising:
receiving one or more multiparty videos representing a conversation between at least a first user and a second user, each multiparty video including acoustic data and video data;
generating conversation features based on the one or more multiparty videos by:
segmenting the one or more videos into multiple utterances;
identifying, for each of the one or more utterances, data for multiple modalities; and
extracting conversation features, for each particular utterance of the one or more utterances, from each of the data for the multiple modalities associated with that particular utterance;
generating conversation analysis indicators, comprising one or more scores for the conversation, by applying the conversation features to a machine learning system;
applying a mapping of the conversation analysis indicators to inferences or actions to determine inference or action results; and
reacting to the conversation according to the inference or action results mapped to the conversation analysis indicators in the mapping.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more conversation analysis indicators comprise one or more conversation impact scores indicating a projected impact of the conversation on at least one of the first user and/or second user.

### Claim 3 (depends on 1)

The method of claim 1,
wherein generating the conversation analysis indicators comprises:
receiving one or more conversation scores from the machine learning system; and
determining a progress score that is a combination of A) the one or more conversation scores and B) one or more previous conversation scores determined for previous conversations involving at least one of the first user and/or the second user;
wherein the mapping includes a mapping of progress scores being below a threshold to an action including transmitting an alert of a low-quality pairing between the first user and the second user; and
wherein the reacting to the conversation includes causing the alert to be transmitted with a control to initiate selection of a new pairing for the first or the second user.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the conversation analysis indicators comprises:
receiving one or more conversation scores from the machine learning system; and
determining a progress score that is a combination of A) the one or more conversation scores and B) one or more previous conversation scores determined for previous conversations involving at least one of the first user and/or the second user.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
retrieving a first user profile for the first user, the first user profile including a progress score;
updating the progress score based on the conversation analysis indicators; and
generating data for a user interface including:
a conversation score benchmark indicating a comparison between the conversation analysis indicators and a reference score; and
the updated progress score in relation to a goal score.

### Claim 6 (depends on 1)

The method of claim 1 further comprising:
retrieving a first user profile for the first user, the first user profile including a progress score;
updating the progress score based on the conversation analysis indicators; and
generating data for a user interface including:
the updated progress score in relation to a goal score; and
a progress score chart indicating historical progress the first user has made toward having the progress score reaching the goal score.

### Claim 7 (depends on 1)

The method of claim 1 further comprising:
retrieving a first user profile for the first user, the first user profile including a progress score;
updating the progress score based on the conversation analysis indicators; and
generating data for a user interface including:
the updated progress score in relation to a goal score; and
at least one sub-score from the conversation analysis indicators including one or more of an engagement score, an ownership score, an openness score, or any combination thereof.

### Claim 8 (depends on 1)

The method of claim 1, wherein the conversation analysis indicators include a series of instant scores comprising scores determined, by the machine learning system, at multiple points during the duration of the conversation, and wherein the method further comprises:
identifying a pattern or quality in the series of instant scores; and
in response causing, in real-time as the conversation progresses, an alert to be provided to the first user and/or the second user with a notification relating to the identified pattern or quality.

### Claim 9 (depends on 1)

The method of claim 1, wherein the conversation analysis indicators include a series of instant scores comprising scores determined, by the machine learning system, at multiple points during the duration of the conversation, and wherein the method further comprises:
identifying a subsegment of the one or more multiparty videos based on an identified pattern or quality in the series of instant scores;
automatically assigning a training moment label to the identified subsegment; and
causing an indication of the subsegment to be transmitted to the first user and/or the second user with an indication of the training moment label.

### Claim 10 (depends on 1)

The method of claim 1, wherein the conversation analysis indicators include a series of instant scores comprising scores determined, by the machine learning system, at multiple points during the duration of the conversation, and wherein the method further comprises:
identifying multiple of training moment subsegments, each training moment subsegment further labeled as a high point or low point by applying a machine learning model trained to synthesize the instant scores; and
causing an indication of the multiple of training moment subsegments to be transmitted to the first user and/or second user with an indication high point or low point labels.

### Claim 11 (depends on 1)

The method of claim 1, wherein the conversation analysis indicators include a series of instant scores comprising scores determined, by the machine learning system, at multiple points during the duration of the conversation.

### Claim 12 (depends on 1)

The method of claim 1,
wherein the one or more conversation analysis indicators includes a staying on goal metric, an interruptions metric, and a time spent listening metric; and
wherein the mapping of conversation analysis indicators to inferences or actions includes mapping the conversation values to a coaching score.

### Claim 13 (depends on 1)

The method of claim 1, wherein the mapping includes a mapping of the conversation analysis indicators to a match inference defining an effectiveness of the pairing between the first user and the second user.

### Claim 14

A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
receiving one or more multiparty videos representing a conversation between at least a first user and a second user;
generating conversation features for at least two modalities of the one or more multiparty videos;
generating conversation analysis indicators, comprising one or more scores for the conversation, by applying the conversation features to a machine learning system, by:
applying the conversation features from each of the at least two modalities, for identified utterances of the one or more multiparty videos, to a corresponding processing part of the machine learning system, to generate utterance output for that utterance; and
combining multiple utterances outputs for corresponding utterances and applying, to a sequential part of the machine learning system, data based on the combined utterance outputs to generate the conversation analysis indicators;
applying a mapping of the conversation analysis indicators to inferences or actions to determine inference or action results; and
reacting to the conversation according to the inference or action results mapped to the conversation analysis indicators in the mapping.

### Claim 15 (depends on 14)

The computer-readable storage medium of claim 14, wherein the one or more conversation analysis indicators comprise one or more conversation impact scores indicating a projected impact of the conversation on at least one of the first user and/or the second user.

### Claim 16 (depends on 14)

The computer-readable storage medium of claim 14, wherein generating the conversation analysis indicators comprises:
receiving one or more conversation scores from the machine learning system; and
determining a progress score that is a combination of A) the one or more conversation scores and B) one or more previous conversation scores determined for previous conversations involving at least one of the first user and/or the second user.

### Claim 17 (depends on 14)

The computer-readable storage medium of claim 14, wherein the operations further comprise:
retrieving a first user profile for the first user, the first user profile including a progress score;
updating the progress score based on the conversation analysis indicators; and
generating data for a user interface including:
a conversation score benchmark indicating a comparison between the conversation analysis indicators and a reference score; and
the updated progress score in relation to a goal score.

### Claim 18 (depends on 14)

The computer-readable storage medium of claim 14, wherein the conversation analysis indicators include a series of instant scores comprising scores determined, by the machine learning system, at multiple points during the duration of the conversation, and wherein the operations further comprise:
identifying a subsegment of the one or more multiparty videos based on an identified pattern or quality in the series of instant scores;
automatically assigning a training moment label the identified subsegment; and
causing an indication of the subsegment to be transmitted to the first user and/or the second user with an indication of the training moment label.

### Claim 19

A computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising:
receiving one or more multiparty videos representing a conversation between at least a first user and a second user, each multiparty video including acoustic data and video data;
generating conversation analysis indicators, comprising one or more scores for the conversation, by applying conversation features based on the one or more multiparty videos to a machine learning system;
applying a mapping of the conversation analysis indicators to inferences or actions to determine inference or action results; and
reacting to the conversation according to the inference or action results mapped to the conversation analysis indicators in the mapping.

### Claim 20 (depends on 19)

The computing system of claim 19, wherein the conversation analysis indicators include a series of instant scores comprising scores determined, by the machine learning system, at multiple points during the duration of the conversation, and wherein the operations further comprise:
identifying a pattern or quality in the series of instant scores; and
in response causing, in real-time as the conversation progresses, an alert to be provided to the first and/or the second user with a notification relating to the identified pattern or quality.
