# US 12,119,028 B2 · Video segment selection and editing using transcript interactions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,119,028 B2
- Document ID: US12119028B2
- Status: Granted
- Kind: B2
- Application number: 17/967,364
- Filed: 2022-10-17
- Published: 2024-04-25
- Granted: 2024-10-15
- Earliest claimed: 2022-10-17
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Xue Bai; Justin Jonathan Salamon; Aseem Omprakash Agarwala; Hijung Shin; Haoran Cai; Joel Richard Brandt; Lubomira Assenova Dontcheva; Cristin Ailidh Fraser
- Web page: https://patentgrep.com/patent/US12119028B2

## CPC

- G11B 27/036: Insert-editing
- G06F 40/166: Editing, e.g. inserting or deleting
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/57: for processing of video signals
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence)
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0485: Scrolling or panning
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Embodiments of the present invention provide systems, methods, and computer storage media for identifying candidate boundaries for video segments, video segment selection using those boundaries, and text-based video editing of video segments selected via transcript interactions. In an example implementation, boundaries of detected sentences and words are extracted from a transcript, the boundaries are retimed into an adjacent speech gap to a location where voice or audio activity is a minimum, and the resulting boundaries are stored as candidate boundaries for video segments. As such, a transcript interface presents the transcript, interprets input selecting transcript text as an instruction to select a video segment with corresponding boundaries selected from the candidate boundaries, and interprets commands that are traditionally thought of as text-based operations (e.g., cut, copy, paste) as an instruction to perform a corresponding video editing operation using the selected video segment.

## Claims

### Claim 1

One or more computer storage media storing computer-useable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
loading, into a video editing interface, a video associated with a representation of a video segmentation that segments the video at boundaries associated with transcript segments of a transcript of the video;
interpreting a first input, interacting with a visualization of the transcript segments and identifying selected transcript text via the video editing interface, as an instruction to select a video segment of the video segmentation corresponding to the selected transcript text; and
interpreting a second input, based on interacting with the visualization of the transcript segments via the video editing interface, as a corresponding instruction to perform a video editing operation comprising cut, copy, or paste on the video segment corresponding to the selected transcript text.

### Claim 2 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising causing generation of the representation of the video segmentation based on extracting sentence segments from the transcript and retiming sentence boundaries, for one or more of the sentence segments that are adjacent to a gap, into the gap based on voice or audio activity in an audio track of the video.

### Claim 3 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising causing generation of the representation of the video segmentation based on subdividing sentence segments to constituent word segments using timestamps extracted from the transcript.

### Claim 4 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising causing generation of the representation of the video segmentation based on extracting word segments from the transcript, and retiming word boundaries, for one or more of the word segments that are adjacent to a gap, into the gap based on voice or audio activity in an audio track of the video.

### Claim 5 (depends on 1)

The one or more computer storage media of claim 1, wherein the first input interacting with the visualization of the transcript segments is a click or tap and drag along the visualization of the transcript segments.

### Claim 6 (depends on 1)

The one or more computer storage media of claim 1, wherein the first input interacting with the visualization of the transcript segments further snaps a selection of transcript text to word boundaries represented by the video segmentation.

### Claim 7 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising, based at least on the first input interacting with the visualization of the transcript segments, causing presentation of a menu that includes an option to execute the video editing operation.

### Claim 8 (depends on 1)

The one or more computer storage media of claim 1, wherein the second input interacting with the visualization of the transcript segments is selected from text-based operations of the selected transcript text comprising cut, copy, paste, or delete to perform on the video segment corresponding to the selected transcript text.

### Claim 9 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising accepting input selecting multiple videos from a media library, generating a corresponding representation of a composite video that combines the multiple videos, and loading the representation of the composite video into the video editing interface, wherein the transcript transcribes audio tracks of the composite video, and the visualization of the transcript is in the video editing interface.

### Claim 10 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising stopping an auto-scroll of the visualization of the transcript segments in response to input scrolling away from an active portion of the transcript, and resuming the auto-scroll in response to input scrolling back to the active portion of the transcript.

### Claim 11 (depends on 1)

The one or more computer storage media of claim 1, the operations further comprising accepting input creating an organizational heading within the transcript without editing the video, and providing an outline view that navigates to a corresponding portion of the transcript in response to input selecting the organizational heading.

### Claim 12

A method comprising:
triggering generation of a representation of a video segmentation that segments a video at boundaries of word segments of a transcript of the video;
interpreting input selecting text of the transcript as an instruction to snap a selection of a corresponding video segment of the video segmentation to a set of the boundaries corresponding to the selected text; and
interpreting a command received via interaction with the selected text as a corresponding instruction to perform a video editing operation comprising cut, copy, or paste on the corresponding video segment.

### Claim 13 (depends on 12)

The method of claim 12, wherein the generation of the representation of the video segmentation comprises extracting sentence segments from the transcript and retiming sentence boundaries, for one or more of the sentence segments that are adjacent to a gap, into the gap based on voice or audio activity in an audio track of the video.

### Claim 14 (depends on 12)

The method of claim 12, wherein the generation of the representation of the video segmentation comprises subdividing sentence segments to the word segments using timestamps extracted from the transcript.

### Claim 15 (depends on 12)

The method of claim 12, wherein the generation of the representation of the video segmentation comprises extracting the word segments from the transcript, and retiming the boundaries, for one or more of the word segments that are adjacent to a gap, into the gap based on voice or audio activity in an audio track of the video.

### Claim 16 (depends on 12)

The method of claim 12, wherein the input selecting the text of the transcript is a click or tap and drag along the text of the transcript.

### Claim 17 (depends on 12)

The method of claim 12, wherein the input selecting the text of the transcript snaps a corresponding selection of the text to start and stop boundaries selected from the boundaries of the word segments.

### Claim 18 (depends on 12)

The method of claim 12, further comprising, based at least on the input selecting the text of the transcript, causing presentation of a menu that includes an option representing the command to execute the video editing operation.

### Claim 19 (depends on 12)

The method of claim 12, wherein the command received via the interaction with the selected text comprises a corresponding selection from text-based operations comprising cut, copy, paste, or delete to perform on the corresponding video segment.

### Claim 20

A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
a video selection tool configured to load, into a video editing interface, a video associated with a video segmentation that segments the video at boundaries of word segments of a transcript of the video; and
a transcript tool of the video editing interface configured to cause presentation of transcript text of the transcript, interpret a first input selecting text of the transcript as an instruction to select a video segment of the video segmentation corresponding to the selected text of the transcript, and interpret a second input interacting with the presentation of the transcript text as a corresponding instruction to perform a video editing operation comprising cut, copy, or paste on the video segment corresponding to the selected text.