# US 2024/0202239 A1 · Transcript aggregaton for non-linear editors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202239 A1
- Document ID: US20240202239A1
- Status: Published application
- Kind: A1
- Application number: 18/082,987
- Filed: 2022-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Nikhil Kalra; Nico Alexander Becherer
- Web page: https://patentgrep.com/patent/US20240202239A1

## CPC

- G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)}
- G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data}
- G06F 16/65: Clustering; Classification
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and computer storage media are disclosed for transcript aggregation. In an example implementation, an audio mix processing map is generated representing an audio signal flow defined by a multi-track sequence that mixes audio sources arranged in audio tracks into a final mix. Using the audio mix processing map, an aggregated audio track is generated that represents which of the audio sources contribute to the final mix during which ranges of time. The aggregated audio track is transformed into an aggregated transcript track by associating transcript segments from transcripts of the audio sources with corresponding ranges of the ranges of time.

## Claims

### Claim 1

One or more non-transitory computer storage media storing computer-useable instructions that, when executed by one or more processing devices, cause the one or more computing processing to perform operations comprising:
generating an audio mix processing map representing an audio signal flow defined by a multi-track sequence that mixes audio sources arranged in audio tracks into a final mix;
generating, using the audio mix processing map, an aggregated audio track that represents which of the audio sources contribute to the final mix during which ranges of time; and
transforming the aggregated audio track into an aggregated transcript track by associating transcript segments from transcripts of the audio sources with corresponding ranges of the ranges of time.

### Claim 2 (depends on 1)

The one or more non-transitory computer storage media of claim 1, wherein the audio mix processing map is a directed graph with a clip node that represents an audio source of the audio sources present at a particular time of the multi-track sequence and a map of duration of the audio source versus volume.

### Claim 3 (depends on 1)

The one or more non-transitory computer storage media of claim 1, wherein the audio mix processing map is a directed graph with an edit node that represents edits applied to an audio source of the audio sources present at a particular time of the multi-track sequence as a map of keyframes of the audio source versus volume.

### Claim 4 (depends on 1)

The one or more non-transitory computer storage media of claim 1, wherein generating the aggregated audio track comprises generating or updating a linked list with entries that represent the audio sources that contribute to the final mix and the ranges of time.

### Claim 5 (depends on 1)

The one or more non-transitory computer storage media of claim 1, the operations further comprising detecting an edit to one of the audio tracks, wherein generating the aggregated audio track and transforming the aggregated audio track into the aggregated transcript track are in response to detecting the edit.

### Claim 6 (depends on 1)

The one or more non-transitory computer storage media of claim 1, wherein generating the aggregated audio track comprises determining whether a particular audio source of the audio sources contributes to the final mix by evaluating metadata represented by the audio mix processing map against one or more contribution tracking rules.

### Claim 7 (depends on 1)

The one or more non-transitory computer storage media of claim 1, wherein generating the aggregated audio track comprises applying one or more contribution tracking rules that evaluate at least one of track mute status, track solo status, volume automation, track item volume, track item gain, or track item classification against metadata represented by the audio mix processing map.

### Claim 8 (depends on 1)

The one or more non-transitory computer storage media of claim 1, the operations further comprising transcribing the audio sources upon importing into the multi-track sequence or in response to a permissions change, wherein transforming the aggregated audio track into the aggregated transcript track is without re-transcribing the audio sources.

### Claim 9 (depends on 1)

The one or more non-transitory computer storage media of claim 1, wherein generating the aggregated audio track comprises determining that a signal level for an audio source of the audio sources falls below a threshold level during a time range, and transforming the aggregated audio track into the aggregated transcript track excludes from the aggregated transcript track a portion of a transcript of the audio source corresponding to the time range in which the signal level falls below the threshold level.

### Claim 10 (depends on 1)

The one or more non-transitory computer storage media of claim 1, the operations further comprising causing an interface of a video or audio editor to present a representation of the transcript segments of the aggregated transcript track or executing a keyword search against the transcript segments of the aggregated transcript track.

### Claim 11

A method comprising:
generating an audio mix processing map representing a level of an audio signal flow through audio track items arranged in audio tracks into a final mix;
generating, using the audio mix processing map, an aggregated audio track that represents contributing audio track items, of the audio track items, that contribute to the final mix; and
transforming the aggregated audio track into an aggregated transcript track by associating transcript segments from transcripts of the audio track items with corresponding ranges of the contributing audio track items.

### Claim 12 (depends on 11)

The method of claim 11, wherein the audio mix processing map is a directed graph with a clip node that represents an audio track item of the audio track items present at a particular time on the audio tracks and a map of duration of the audio track item versus volume.

### Claim 13 (depends on 11)

The method of claim 11, wherein the audio mix processing map is a directed graph with an edit node that represents edits applied to an audio track item of the audio track items present at a particular time on the audio tracks as a map of keyframes of the audio track item versus volume.

### Claim 14 (depends on 11)

The method of claim 11, wherein generating the aggregated audio track comprises generating or updating a linked list with entries that represent the contributing audio track items and the corresponding ranges.

### Claim 15 (depends on 11)

The method of claim 11, further comprising detecting an edit to one of the audio tracks, wherein generating the aggregated audio track and transforming the aggregated audio track into the aggregated transcript track are in response to detecting the edit.

### Claim 16 (depends on 11)

The method of claim 11, wherein generating the aggregated audio track comprises determining whether a particular audio track item of the audio track items contributes to the final mix by evaluating metadata represented by the audio mix processing map against one or more contribution tracking rules.

### Claim 17 (depends on 11)

The method of claim 11, wherein generating the aggregated audio track comprises applying one or more contribution tracking rules that evaluate at least one of track mute status, track solo status, volume automation, track item volume, track item gain, or track item classification against metadata represented by the audio mix processing map.

### Claim 18 (depends on 11)

The method of claim 11, further comprising transcribing the audio track items upon importing into the audio tracks or in response to a permissions change, wherein transforming the aggregated audio track into the aggregated transcript track is without re-transcribing the audio track items.

### Claim 19 (depends on 11)

The method of claim 11, wherein generating the aggregated audio track comprises determining that a signal level for an audio track item of the audio track items falls below a threshold level during a time range, and transforming the aggregated audio track into the aggregated transcript track excludes from the aggregated transcript track a portion of a transcript of the audio track item corresponding to the time range in which the signal level falls below the threshold level.

### Claim 20

A system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
an aggregator configured to:
identify, from audio track items arranged in a multi-track sequence that mixes the audio track items into a final mix, contributing audio track items that contribute an audio signal to the final mix;
generate an aggregated audio track that represents the contributing audio track items; and
execute an operation using the aggregated audio track.
