# US 2021/0064327 A1 · Audio highlighter

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0064327 A1
- Document ID: US20210064327A1
- Status: Published application
- Kind: A1
- Application number: 16/550,776
- Filed: 2019-08-26
- Published: 2021-03-04
- Earliest claimed: 2019-08-26
- Inventors: Abigail Ispahani
- Web page: https://patentgrep.com/patent/US20210064327A1

## CPC

- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system and method for processing digital audio data, transcribing spoken word audio content from the digital audio data into text data, associating the text data with the digital audio data, reviewing and organizing the transcribed text, and playing back selected portions of the digital audio data associated with the transcribed text is presented. In one or more embodiments, the present invention allows a listener to mark and transcribe audio passages in, for example, a podcast or audio book, for later searching and/or reference. Thus, by analogy to use of a highlighter pen with printed text, the present invention provides an “audio highlighter” for spoken words.

## Claims

### Claim 1

A method for providing audio highlighter functionality comprising the steps of:
receiving a digital audio stream synchronously from a digital audio playback application;
starting a timer that measures a current playback position in the digital audio stream;
creating a log file associated with the digital audio stream; and
transcribing the digital audio stream to text;
wherein the step of transcribing the digital audio stream to text comprises the substeps of:
dividing the digital audio stream into a plurality of digital audio chunks;
associating a unique timestamp with each digital audio chunk;
converting each digital audio chunk into a corresponding text string;
associating the unique timestamp of each digital audio chunk to the corresponding text string; and
recording each text string and its associated unique timestamp to the log file.

### Claim 2 (depends on 1)

The method of claim 1 wherein the step of transcribing the digital audio stream to text is started in response to user input.

### Claim 3 (depends on 2)

The method of claim 2 wherein the step of transcribing the digital audio stream to text is stopped in response to user input.

### Claim 4 (depends on 1)

The method of claim 1 further comprising the step of providing a first user interface to display a graphical timeline representation of the digital audio stream, wherein the graphical timeline representation comprises at least one highlight mark indicating a position in the digital audio stream of the unique timestamp associated with the corresponding text string.

### Claim 5 (depends on 4)

The method of claim 4 further comprising the step of starting playback of the digital audio stream from one of the unique timestamps in response to user selection of the corresponding highlight mark in the first user interface.

### Claim 6 (depends on 1)

The method of claim 1 further comprising the step of providing a second user interface to display the at least one text string and its associated unique timestamp.

### Claim 7 (depends on 6)

The method of claim 6 further comprising the step of starting playback of the digital audio stream from one of the unique timestamps in response to user selection of the corresponding text string in the second user interface.

### Claim 8 (depends on 1)

The method of claim 1 wherein the step of converting each digital audio chunk into a corresponding text string comprises the substeps of:
sending the digital audio chunk to a speech-to-text converter;
transcribing the digital audio chunk into its corresponding text string with the speech-to-text converter; and
receiving the text string from the speech-to-text converter.

### Claim 9 (depends on 8)

The method of claim 8 wherein the speech-to-text converter is located on a server computer system, wherein the step of transcribing the digital audio chunk into its corresponding text string with the speech-to-text converter is performed by the server computer system, and wherein the remaining method steps are performed by a mobile device.

### Claim 10

An audio highlighter system comprising:
a microprocessor;
a memory;
computer-readable instructions stored in the memory and executing on the microprocessor; and
digital audio data stored in the memory;
wherein the audio highlighter system is configured to, in accordance with the computer readable instructions:
begin playback of the digital audio data;
start a timer that measures a current playback position in the digital audio data;
create a log file associated with the digital audio data; and
transcribe the digital audio stream to text by dividing the digital audio data into a plurality of digital audio chunks, associating a unique timestamp with each digital audio chunk, converting each digital audio chunk into a corresponding text string, associating the unique timestamp of each digital audio chunk to the corresponding text string, and recording each text string and its associated unique timestamp to the log file.

### Claim 11 (depends on 10)

The audio highlighter system of claim 10 wherein the audio highlighter system is further configured to start the transcription of the digital audio stream in response to user input.

### Claim 12 (depends on 10)

The audio highlighter system of claim 10 wherein the audio highlighter system is further configured to stop the transcription of the digital audio stream in response to user input.

### Claim 13 (depends on 10)

The audio highlighter system of claim 10 wherein the audio highlighter system is further configured to provide a first user interface to display a graphical timeline representation of the digital audio stream, wherein the graphical timeline representation comprises at least one highlight mark indicating a position in the digital audio stream of the unique timestamp associated with the corresponding text string.

### Claim 14 (depends on 13)

The audio highlighter system of claim 13 wherein the audio highlighter system is further configured to start playback of the digital audio stream from one of the unique timestamps in response to user selection of the corresponding highlight mark in the first user interface.

### Claim 15 (depends on 10)

The audio highlighter system of claim 10 wherein the audio highlighter system is further configured to provide a second user interface to display the at least one text string and its associated unique timestamp.

### Claim 16 (depends on 15)

The audio highlighter system of claim 15 wherein the audio highlighter system is further configured to start playback of the digital audio stream from one of the unique timestamps in response to user selection of the corresponding text string in the second user interface.

### Claim 17 (depends on 10)

The audio highlighter system of claim 10 further comprising a speech-to-text converter, wherein the audio highlighter system is further configured to convert each digital audio chunk into a corresponding text string by sending the digital audio chunk to the speech-to-text converter for transcription and receiving the transcribed text string from the speech-to-text converter.

### Claim 18 (depends on 17)

The audio highlighter system of claim 17 wherein the speech-to-text converter is located on a server computer system, and wherein the transcription of the digital audio chunk into its corresponding text string with the speech-to-text converter is performed by the server computer system.

### Claim 19

A method for providing audio highlighter functionality comprising the steps of:
receiving a digital audio stream synchronously from a digital audio playback application;
starting a timer that measures a current playback position in the digital audio stream;
creating a log file associated with the digital audio stream;
transcribing the digital audio stream to text in response to user input;
providing a first user interface to display a graphical timeline representation of the digital audio stream, wherein the graphical timeline representation comprises at least one highlight mark indicating a position in the digital audio stream of the unique timestamp associated with the corresponding text string;
starting playback of the digital audio stream from one of the unique timestamps in response to user selection of the corresponding highlight mark in the first user interface;
providing a second user interface to display the at least one text string and its associated unique timestamp; and
starting playback of the digital audio stream from one of the unique timestamps in response to user selection of the corresponding text string in the second user interface;
wherein the step of transcribing the digital audio stream to text comprises the substeps of:
dividing the digital audio stream into a plurality of digital audio chunks;
associating a unique timestamp with each digital audio chunk;
converting each digital audio chunk into a corresponding text string;
associating the unique timestamp of each digital audio chunk to the corresponding text string; and
recording each text string and its associated unique timestamp to the log file; and
wherein the step of converting each digital audio chunk into a corresponding text string comprises the substeps of:
sending the digital audio chunk to a speech-to-text converter located on a server computer system;
transcribing the digital audio chunk into its corresponding text string with the speech-to-text converter on the server computer system; and
receiving the text string from the speech-to-text converter.
