# US 11,308,331 B2 · Multimedia content summarization method and system thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,331 B2
- Document ID: US11308331B2
- Status: Granted
- Kind: B2
- Application number: 16/796,069
- Filed: 2020-02-20
- Published: 2021-07-01
- Granted: 2022-04-19
- Earliest claimed: 2019-12-31
- Assignees: Wipro Limited
- Applicants: Wipro Limited
- Inventors: Manjunath Ramachandra Iyer
- Web page: https://patentgrep.com/patent/US11308331B2

## CPC

- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/47: {Detecting features for summarising video content}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- H04N 21/2335: {involving reformatting operations of audio signals, e.g. by converting from one coding standard to another (details of audio signal transcoding G10L19/173)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/2402: {Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876)}
- H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/8547: involving timestamps for synchronizing content
- G06V 20/44: {Event detection}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A method and system for summarizing multimedia content is disclosed. The method includes the steps of extracting a set of video files from a multimedia content such that each of the set of video files comprises a plurality of frames, and summarizing each of the set of video files to generate a set of summarized video files. Summarizing includes modifying a number of frames in each of the set of video files while retaining a caption generated for each of the set of video files. The method further includes generating sets of bridge frames for the set of summarized video files, based on a deep learning model. A set of bridge frames from the sets of bridge frames maintains continuity between corresponding adjacent summarized video files. The method includes generating a summarized multimedia content based on the set of summarized video files and the sets of bridge frames.

## Claims

### Claim 1

A method for summarizing multimedia content, the method comprising:
extracting, by a content summarization device, a set of video files from a multimedia content, wherein each of the set of video files comprises a plurality of frames;
summarizing, by the content summarization device, each of the set of video files to generate a set of summarized video files, wherein summarizing the set of video files comprises modifying a number of frames in each of the set of video files, and wherein modifying the number of frames in a video file from the set of video files retains a caption generated for the video file;
generating, by the content summarization device, sets of bridge frames for the set of summarized video files, based on a deep learning model, wherein a set of bridge frames from the sets of bridge frames maintains continuity between corresponding adjacent summarized video files; and
generating, by the content summarization device, a summarized multimedia content based on the set of summarized video files and the sets of bridge frames.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
retrieving the multimedia content from one of a plurality of sources; and
splitting the multimedia content into audio content and video content based on a time stamp mapping technique.

### Claim 3 (depends on 2)

The method of claim 2, wherein extracting the set of video files comprises:
iteratively determining at least one change in context of the video content based on comparison of adjacent frames within the video content, wherein the change in context corresponds to difference between adjacent frames being greater than a predefined threshold; and
extracting a video file from the video content at each of the at least one change in context in the video content to create the set of video files.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
extracting a set of audio files from the audio content; and
summarizing each of the set of audio files to generate a set of summarized audio files, wherein summarizing an audio file from the set of audio files comprises:
converting each the audio file into text;
parsing the text associated with the audio file to identify at least one subject and interaction of the at least one subject with at least one object, based on natural language processing techniques; and
summarizing the audio file based on the identification of the least one subject and interaction of the at least one subject with the at least one object.

### Claim 5 (depends on 4)

The method of claim 4, further comprising synchronizing each of the set of summarized audio files with a corresponding summarized video file from the set of summarized video files, wherein a summarized audio file from the set of summarized audio files defines boundary of a corresponding summarized video file.

### Claim 6 (depends on 1)

The method of claim 1, wherein summarizing each of the set of video files to generate a set of summarized video files comprises further comprising:
determining a plurality of congestion categories for a communication channel, wherein each of the plurality of congestion categories correspond to at least one congestion status of the communication channel;
evaluating, for each of the plurality of congestion categories, a degree of summarization required for each of the set of video files; and
generating, for each of the plurality of congestion categories, the set of summarized video files based on the associated degree of summarization.

### Claim 7 (depends on 6)

The method of claim 6 further comprising rendering the summarized multimedia content on a device associated with a user.

### Claim 8 (depends on 7)

The method of claim 7, wherein rendering comprises:
determining a plurality of network parameters of a communication channel being accessed by the device;
determining a Random Early Detection (RED) value of the communication channel based on the plurality of network parameters;
determining a congestion status of the communication channel based on the determined RED value using the deep learning model;
mapping the congestion status with at least one of the plurality of congestion categories; and
rendering the summarized multimedia content based on the mapped congestion category.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the summarized multimedia content comprises interleaving each of the sets of bridge frames between the corresponding adjacent summarized video files from the set of summarized video files.

### Claim 10 (depends on 1)

The method of claim 1, wherein modifying the number of frames in a video file from the set of video files comprises at least one of:
iteratively removing at least one frame starting from the last frame in the video file, wherein the at least one frame are removed iteratively till the caption associated with the video file is maintained; and
iteratively adding at least one frame to create the video file, wherein the at least one frame is iteratively added till the caption associated with the video file is generated.

### Claim 11 (depends on 1)

The method of claim 1, wherein the deep learning model comprises at least one of a multilayer perceptron (MLP) model, a long short-term memory (LSTM) model, a convolutional neural network (CNN) model, a recursive neural network (RNN) model, or a recurrent neural network (RNN) model.

### Claim 12

A method for summarizing multimedia content, the method comprising:
extracting, by a content summarization device, a set of video files and a corresponding set of audio files from a multimedia content, wherein each of the set of video files comprises a plurality of frames;
summarizing, by the content summarization device, each of the set of video files to generate a set of summarized video files and each of the set of audio files to generate a set of summarized audio files,
wherein summarizing the set of video files comprises modifying a number of frames in each of the set of video files, and wherein modifying the number of frames in a video file from the set of video files retains a caption generated for the video file, and
wherein summarizing an audio file from the set of audio files comprises:
converting each the audio file into text;
parsing the text associated with the audio file to identify at least one subject and interaction of the at least one subject with at least one object, based on natural language processing techniques; and
summarizing the audio file based on the identification of the least one subject and interaction of the at least one subject with the at least one object;
synchronizing, by the content summarization device, each of the set of summarized audio files with a corresponding summarized video file from the set of summarized video files, wherein a summarized audio file from the set of summarized audio files defines boundary of a corresponding summarized video file.

### Claim 13 (depends on 12)

The method of claim 12, further comprising adding at least one frame to a summarized video file from the set of summarized video file to match boundary of a corresponding summarized audio file from the set of summarized audio files.

### Claim 14

A system for summarizing multimedia content, the system comprising:
a processor; and
a memory communicatively coupled to the processor, wherein the memory stores processor instructions, which, on execution, causes the processor to:
extract a set of video files from a multimedia content, such that each of the set of video files comprises a plurality of frames;
summarize each of the set of video files to generate a set of summarized video files, such that summarizing the set of video files comprises modifying a number of frames in each of the set of video files, and wherein modifying the number of frames in a video file from the set of video files retains a caption generated for the video file;
generate sets of bridge frames for the set of summarized video files, based on a deep learning model, such that a set of bridge frames from the sets of bridge frames maintains continuity between corresponding adjacent summarized video files; and
generate a summarized multimedia content based on the set of summarized video files and the sets of bridge frames.

### Claim 15 (depends on 14)

The system of claim 14, wherein the processor instructions further cause the processor to:
retrieve the multimedia content from one of a plurality of sources; and
split the multimedia content into audio content and video content based on a time stamp mapping technique.

### Claim 16 (depends on 15)

The system of claim 15, wherein the processor instructions further cause the processor to:
iteratively determine at least one change in context of the video content based on comparison of adjacent frames within the video content, such that the change in context corresponds to difference between adjacent frames being greater than a predefined threshold; and
extract a video file from the video content at each of the at least one change in context in the video content to create the set of video files.

### Claim 17 (depends on 15)

The system of claim 15, wherein the processor instructions further cause the processor to:
extract a set of audio files from the audio content; and
summarize each of the set of audio files to generate a set of summarized audio files, wherein summarizing an audio file from the set of audio files comprises:
convert each the audio file into text;
parse the text associated with the audio file to identify at least one subject and interaction of the at least one subject with at least one object, based on natural language processing techniques; and
summarize the audio file based on the identification of the least one subject and interaction of the at least one subject with the at least one object.

### Claim 18 (depends on 15)

The system of claim 15, wherein the processor instructions further cause the processor to synchronize each of the set of summarized audio files with a corresponding summarized video file from the set of summarized video files, wherein a summarized audio file from the set of summarized audio files defines boundary of a corresponding summarized video file.

### Claim 19 (depends on 14)

The system of claim 14, wherein the processor instructions further cause the processor to:
determine a plurality of congestion categories for a communication channel, wherein each of the plurality of congestion categories correspond to at least one congestion status of the communication channel;
evaluate, for each of the plurality of congestion categories, a degree of summarization required for each of the set of video files; and
generate, for each of the plurality of congestion categories, the set of summarized video files based on the associated degree of summarization.

### Claim 20 (depends on 19)

The system of claim 19, wherein the processor instructions further cause the processor to render the summarized multimedia content on a device associated with a user.

### Claim 21 (depends on 20)

The system of claim 20, wherein the processor instructions further cause the processor to:
determine a plurality of network parameters of a communication channel being accessed by the device;
determine a Random Early Detection (RED) value of the communication channel based on the plurality of network parameters;
determine a congestion status of the communication channel based on the determined RED value using the deep learning model;
map the congestion status with at least one of the plurality of congestion categories; and
render the summarized multimedia content based on the mapped congestion category.

### Claim 22 (depends on 14)

The system of claim 14, wherein generating the summarized multimedia content comprises interleaving each of the sets of bridge frames between the corresponding adjacent summarized video files from the set of summarized video files.

### Claim 23 (depends on 14)

The system of claim 14, wherein the processor instructions further cause the processor to:
iteratively remove at least one frame starting from the last frame in the video file, wherein the at least one frame are removed iteratively till the caption associated with the video file is maintained; and
iteratively add at least one frame to create the video file, wherein the at least one frame is iteratively added till the caption associated with the video file is generated.