# US 2024/0205520 A1 · Method for coherent, unsupervised, transcript-based, extractive summarisation of long videos of spoken content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0205520 A1
- Document ID: US20240205520A1
- Status: Published application
- Kind: A1
- Application number: 18/539,654
- Filed: 2023-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: University College Dublin, National University of Ireland
- Applicants: University College Dublin, National University of Ireland
- Inventors: Ricardo Simon Carbajo; Sean Mahon; Hanene Jemoui
- Web page: https://patentgrep.com/patent/US20240205520A1

## CPC

- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06V 20/47: {Detecting features for summarising video content}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Disclosed is a video summarisation method that includes converting an input video file to a video transcript using speech recognition, segmenting the video transcript into a plurality of paragraphs using a transformer based model, performing relevance ranking of the plurality of paragraphs by assigning a paragraph importance score to each paragraph, based on relevance of each paragraph to an overall content of the video transcript, creating a plurality of candidate summaries from the plurality of paragraphs, each candidate summary comprising two or more paragraphs, performing coherence reranking of the plurality of candidate summaries based on a combination of coherence and relevance of each candidate summary, and selecting a summary based on the coherence reranking.

## Claims

### Claim 1

A video summarisation method comprising:
converting an input video file to a video transcript using speech recognition;
segmenting the video transcript into a plurality of paragraphs using a transformer based model;
performing relevance ranking of the plurality of paragraphs by assigning a paragraph importance score to each paragraph, based on relevance of each paragraph to an overall content of the video transcript;
creating a plurality of candidate summaries from the plurality of paragraphs, each candidate summary comprising two or more paragraphs;
performing coherence reranking of the plurality of candidate summaries based on a combination of coherence and relevance of each candidate summary; and
selecting a summary based on the coherence reranking.

### Claim 2 (depends on 1)

The video summarisation method as claimed in claim 1, wherein the performing the coherence reranking comprises assigning a final score to each candidate summary based on paragraph importance scores of each paragraph of corresponding candidate summary, and an average coherence score of pairs of consecutive paragraphs of corresponding candidate summary.

### Claim 3 (depends on 1)

The video summarisation method as claimed in claim 1, wherein the video transcript includes a plurality of sentences and timestamps.

### Claim 4 (depends on 1)

The video summarisation method as claimed in claim 1, further comprising generating a summary video or audio by identifying segments of the input video corresponding to paragraphs of the selected summary based on the timestamps, and joining the identified segments.

### Claim 5 (depends on 1)

The video summarisation method as claimed in claim 1 further comprising the steps of:
performing relevance ranking of a plurality of sentences of the selected summary based on relevance of each sentence to overall content of the selected summary;
creating a plurality of candidate shortened summaries from the plurality of sentences, each candidate shortened summary comprising two or more sentences;
performing coherence reranking of the plurality of candidate shortened summaries based on a combination of coherence and relevance of each candidate shortened summary;
selecting a shortened summary based on the coherence reranking; and
generating a trailer video based on the shortened summary.

### Claim 6 (depends on 1)

The video summarisation method as claimed in claim 1, wherein the segmenting the video transcript into the plurality of paragraphs comprising constraining length of each paragraph based on corresponding fixed upper and lower time limits in corresponding input audio.

### Claim 7 (depends on 1)

The video summarisation method as claimed in claim 1, wherein the performing the relevance ranking comprises embedding the plurality of paragraphs with pretrained transformer model(s), computing embedded similarities among each pair of paragraphs using a similarity matrix, and using a page rank algorithm to rank the plurality of paragraphs in order of their relevance.

### Claim 8 (depends on 1)

The video summarisation method as claimed in claim 1, further comprising generating the paragraph importance score for each paragraph using an ensemble of a plurality of transformer models, each trained using different unsupervised objectives, wherein the outputs of the plurality of transformer models are combined to generate the paragraph importance score for each paragraph.

### Claim 9 (depends on 1)

The video summarisation method as claimed in claim 1, wherein the transformer based model for segmenting the video transcript into the plurality of paragraphs is trained using unsupervised and supervised domain adaptation, transferring the learning to speech transcripts on specific target domains, and wherein one or more adapter layers inserted into general NLP transformer models are trained using data of other domains to generate the transformer based model for segmenting the video transcript.

### Claim 10 (depends on 1)

The video summarisation method as claimed claim 1, wherein the coherence reranking further comprises generating a coherence score of each pair of paragraphs using a supervised transformer based model, wherein the transformer based model is a Siamese architecture.

### Claim 11 (depends on 1)

The video summarisation method as claimed in claim 1, further comprising:
computing contextualised embeddings of each token of an input pair of paragraphs;
computing scores and weights of each of the m×n pairs of the tokens of a paragraph embedding of length m and width n; and
computing a weighted average of the scores and weights of the pair of paragraphs to generate a coherence score.

### Claim 12 (depends on 1)

The video summarisation method as claimed in claim 1, wherein the coherence score is 1, when the input pair of paragraphs are coherent, and the coherence score is −1, when the input pair of paragraphs are non-coherent.

### Claim 13

A video summarisation system, comprising:
a memory to store one or more instructions; and
a processor communicatively coupled to the memory, and configured to execute the one or more instructions to:
convert an input video file to a video transcript using speech recognition;
segment the video transcript into a plurality of paragraphs using a transformer based model;
perform relevance ranking of the plurality of paragraphs by assigning a paragraph importance score to each paragraph, based on relevance of each paragraph to an overall content of the video transcript;
create a plurality of candidate summaries from the plurality of paragraphs, each candidate summary comprising two or more paragraphs;
perform coherence reranking of the plurality of candidate summaries based on a combination of coherence and relevance of each candidate summary; and
select a summary based on the coherence reranking.
