# US 2024/0205039 A1 · Real-time tone feedback in video conferencing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0205039 A1
- Document ID: US20240205039A1
- Status: Published application
- Kind: A1
- Application number: 18/392,849
- Filed: 2023-12-21
- Published: 2024-06-20
- Earliest claimed: 2022-03-18
- Assignees: Grammarly, Inc.
- Applicants: Grammarly, Inc.
- Inventors: VIPUL RAHEJA; DIMITRIOS ALIKANIOTIS
- Web page: https://patentgrep.com/patent/US20240205039A1

## CPC

- H04L 12/1831: {Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 40/176: {Dynamic expression}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented process is programmed to programmatically receive, using a first computer system, electronic digital data representing input time-correlated speech data and video data, determine a first text sequence corresponding to the input time-correlated speech data, the first text sequence comprising unstructured natural language text, determining syntactic structure data associated with the first text sequence, inputting the time-correlated video data and the syntactic structure data associated with the first text sequence into one or more machine learning models, the machine learning models producing an output of one or more scores for at least a portion of the time-correlated video data and first text sequence, transforming the output of one or more scores to yield and output set of summary points and suggestions, and transmitting a graphical element of the output set of summary points and suggestions for display.

## Claims

### Claim 1

A computer-implemented method comprising:
using a computer system and under stored program control, receiving electronic digital data representing input time-correlated speech data and video data;
by the computer system, determining a first text sequence corresponding to the input time-correlated speech data;
by the computer system, determining a syntactic structure data associated with the first text sequence;
by the computer system, inputting the time-correlated video data and the syntactic structure data associated with the first text sequence into one or more machine-learning models, the machine-learning models having been trained to produce, and producing, an output set of summary points and suggestions;
by the computer system, transmitting a graphical element of the output set of summary points and suggestions to a computing device, wherein rendering the graphical element using presentation functions of the computing device causes displaying the graphical element at the computing device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
by the computer system, presenting a plurality of functions of the output set of summary points and suggestions via a graphical user interface and receiving input via the graphical user interface specifying a plurality of selections of the plurality of functions;
by the computer system, updating the one or more machine-learning models using the plurality of selections, the first text sequence, and the syntactic structure data associated with the first text sequence;
by the computer system, applying the one or more machine-learning models to update the output set of summary points and suggestions;
by the computer system, transmitting a graphical element of the updated output set of summary points and suggestions to a computing device, wherein rendering the graphical element using presentation functions of the computing device causes displaying the graphical element at the computing device.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
by the computer system, inputting the time-correlated video data and the syntactic structure data into one or more machine-learning models, the machine-learning models having been trained to determine a first expression and a second expression; and
by the computer system, transmitting a graphical element of the first and second expressions to a computing device, wherein rendering the graphical element using presentation functions of the computing device causes displaying the graphical element at the computing device.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, the machine-learning models comprising any one or more of expression determination systems and personality impression systems.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, the one or more expression determination systems comprising a video-driven expression system to receive the time-correlated video data, the time-correlated video data having a plurality of frames that depict facial expressions from a video from whom the input time-correlated speech data and video data was obtained.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, the one or more personality impression systems comprising a video-driven impression system to receive the time-correlated video data and an audio driven impression system to receive the time-correlated audio data, the time-correlated video data having a plurality of frames that depict facial expressions from a video from whom the input time-correlated speech data and video data was obtained.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, the output set of summary points and suggestions comprising one or more of a classification of tone, speech, personality, and expression.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising using a digital lexicon to associate the syntactic structure data for the first text sequence with a tone label.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising, before the transmitting, ranking the output set of summary points and suggestions based on a ranking criterion.
