# US 2020/0004803 A1 · Emphasizing key points in a speech file and structuring an associated transcription

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0004803 A1
- Document ID: US20200004803A1
- Status: Published application
- Kind: A1
- Application number: 16/024,212
- Filed: 2018-06-29
- Published: 2020-01-02
- Earliest claimed: 2018-06-29
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Franck Dernoncourt; Walter Wei-Tuh Chang; Seokhwan Kim; Sean Fitzgerald; Ragunandan Rao Malangully; Laurie Marie Byrum; Frederic Thevenet; Carl Iwan Dockhorn
- Web page: https://patentgrep.com/patent/US20200004803A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 40/106: Display of layout of documents; Previewing
- G06F 40/14: Tree-structured documents (parsing G06F40/205; validation G06F40/226)
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G10L 15/18: using natural language modelling
- G10L 25/51: for comparison or discrimination
- G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40)
- G10L 15/16: using artificial neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques are disclosed for generating a structured transcription from a speech file. In an example embodiment, a structured transcription system receives a speech file comprising speech from one or more people and generates a navigable structured transcription object. The navigable structured transcription object may comprise one or more data structures representing multimedia content with which a user may navigate and interact via a user interface. Text and/or speech relating to the speech file can be selectively presented to the user (e.g., the text can be presented via a display, and the speech can be aurally presented via a speaker).

## Claims

### Claim 1

A method for generating a structured transcription of a speech file, the method comprising:
processing said speech file to determine at least one sentence;
processing said at least one sentence to generate a document tree structure comprising a plurality of sections;
converting said speech file to text;
generating a highlighted representation of said text by:
computing term-frequency vectors based on said text;
performing a highlighting operation on each sentence by:
performing a binary classification based upon a maximum term-frequency vector associated with said sentence and acoustic features associated with said sentence;
if said binary classification outputs a pre-determined value, performing a formatting operation to highlight said sentence;
performing an interactive navigation of said document based upon said speech file,
said document tree structure and said highlighted representation.

### Claim 2 (depends on 1)

The method according to claim 1, wherein processing said at least one sentence to generate a document tree structure comprises:
generating a sentence embedding for each sentence; and,
processing said sentence embedding to output a binary value indicating whether said sentence concludes a section.

### Claim 3 (depends on 1)

The method according to claim 1, wherein processing said at least one sentence to generate a document tree structure comprises performing a TextTiling process.

### Claim 4 (depends on 1)

The method according to claim 1, wherein said summary comprises an extractive summary, or an abstractive summary, or both an extractive summary and an abstractive summary.

### Claim 5 (depends on 4)

The method according to claim 4, wherein said extractive summary is utilized to perform text highlighting.

### Claim 6 (depends on 3)

The method according to claim 3, wherein processing said at least one sentence to generate a document tree structure further comprises:
segmenting said at least one sentence into at least one segment;
utilizing said at least one segment to generate at least one section; and,
constructing said document tree from said at least one section.

### Claim 7 (depends on 1)

The method according to claim 1, wherein processing said speech file to determine at least one sentence comprises:
performing an automatic speech recognition (“ASR”) on said speech file to generate a first file;
processing said first file to determine sentences to generate a second file; and,
processing said second file to decompose run-on sentences into smaller logical sentences to generate a third file.

### Claim 8 (depends on 7)

The method according to claim 7, further comprising processing said third file to associate sentences with respective speakers to generate a fourth file.

### Claim 9

A system for processing a speech file, the system comprising:
a sentence identifier, wherein said sentence identifier generates at least one sentence from said speech file;
a summarizer, wherein said summarizer generates at least one summary based upon said at least one sentence;
a document tree analyzer, wherein said document tree analyzer generates a document tree structure;
a document highlighting module for generating a highlighted textual representation of said speech file, wherein said document highlighting module further comprises:
a term-frequency vector computation module for generating term-frequency vectors;
a binary classifier for performing a binary classification of each sentence based upon a maximum term-frequency vector associated with said sentence and acoustic features associated with said sentence; and,
a navigator, wherein said navigator performs an interactive navigation of said document based upon said speech file, said document tree structure and said highlighted textual representation.

### Claim 10 (depends on 9)

The system according to claim 9, wherein each of said at least one summary comprises an abstractive summary and an extractive summary.

### Claim 11 (depends on 9)

The system according to claim 9, wherein said document tree analyzer generates said document tree structure using a TextTiling process.

### Claim 12 (depends on 9)

The system according to claim 9, wherein said sentence identifier comprises:
a speech recognition engine, wherein said speech recognition engine generates a text representation of said speech file;
a sentence boundary detector, wherein said sentence boundary detector generates at least one sentence based upon said text representation;
a run-on sentence detector, wherein said run-on sentence-detector splits a run-on sentence into at least two sentences; and,
a speaker sentence identifier, wherein said speaker sentence identifier associates each of said at last one sentence with a respective speaker.

### Claim 13 (depends on 9)

The system according to claim 9, wherein said document tree analyzer:
segments said at least one sentence into at least one segment;
utilizes said at least one segment to generate at least one section; and,
constructs said document tree from said at least one section.

### Claim 14 (depends on 9)

The system according to claim 9, wherein said document tree analyzer:
generates a sentence embedding for each sentence; and,
processes said sentence embedding to output a binary value indicating whether said sentence concludes a section.

### Claim 15

A computer program product including one or more non-transitory machine-readable mediums encoded with instructions that when executed by one or more processors cause a process to be carried out for processing a speech file, the process comprising:
processing said speech file to determine at least one sentence;
processing said at least one sentence to generate a document tree structure comprising a plurality of sections;
converting said speech file to text;
generating a highlighted representation of said text by:
computing term-frequency vectors based on said text;
performing a highlighting operation on each sentence by:
performing a binary classification based upon a maximum term-frequency vector associated with said sentence and acoustic features associated with said sentence;
if said binary classification outputs a pre-determined value, performing a formatting operation to highlight said sentence;
performing an interactive navigation of said document based upon said speech file,
said document tree and said highlighted representation.

### Claim 16 (depends on 15)

The computer program product according to claim 15, wherein processing said at least one sentence to generate a document tree structure comprises:
generating a sentence embedding for each sentence; and,
processing said sentence embedding to output a binary value indicating whether said sentence concludes a section.

### Claim 17 (depends on 15)

The computer program product according to claim 15, wherein processing said at least one sentence to generate a document tree structure comprises performing a TextTiling process.

### Claim 18 (depends on 15)

The computer program product according to claim 15, wherein said summary comprises an extractive summary, or an abstractive summary, or both an extractive summary and an abstractive summary.

### Claim 19 (depends on 18)

The computer program product according to claim 18, wherein said extractive summary is utilized to perform text highlighting.

### Claim 20 (depends on 15)

The computer program product according to claim 15, wherein processing said speech file to determine at least one sentence further comprises:
performing an automatic speech recognition (“ASR”) on said speech file to generate a first file;
processing said first file to determine sentences to generate a second file; and,
processing said second file to decompose run-on sentences into smaller logical sentences to generate a third file.
