Multimodal video summarization
AssigneeAdobe Inc.
InventorsZhaowen Wang, Trung Bui, Bo He
Life of this patent
Estimating the term…
Abstract
A method includes receiving a video input and a text transcription of the video input. The video input includes a plurality of frames and the text transcription includes a plurality of sentences. The method further includes determining, by a multimodal summarization model, a subset of key frames of the plurality of frames and a subset of key sentences of the plurality of sentences. The method further includes providing a summary of the video input and a summary of the text transcription based on the subset of key frames and the subset of key sentences.
Claims20
A method comprising:
- receiving a video input and a text transcription of the video input, wherein the video input includes a plurality of frames and the text transcription includes a plurality of sentences;
- generating a segment of a plurality of segments, the segment comprising a video embedding of a video frame of the plurality of frames and a text embedding of a sentence of the plurality of sentences, wherein the video embedding and the text embedding comprise positional information associated with the video frame and the sentence respectively, and wherein a number of video embeddings of the segment is based on a duration between a start time and an end time associated with the sentence of the text embedding;
- determining, by a multimodal summarization model that inputs the plurality of segments, a subset of key frames of the plurality of frames and a subset of key sentences of the plurality of sentences;
- and
- providing a summary of the video input and a summary of the text transcription based on the subset of key frames and the subset of key sentences.
A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
- receiving a video input and a text transcription of the video input, wherein the video input includes a plurality of frames and the text transcription includes a plurality of sentences;
- generating a segment of a plurality of segments, the segment comprising a video embedding of a video frame of the plurality of frames and a text embedding of a sentence of the plurality of sentences, wherein the video embedding and the text embedding comprise positional information associated with the video frame and the sentence respectively, and wherein a number of video embeddings of the segment is based on a duration between a start time and an end time associated with the sentence of the text embedding;
- determining, by a multimodal summarization model that inputs the plurality of segments, a subset of key frames of the plurality of frames and a subset of key sentences of the plurality of sentences;
- and
- providing a summary of the video input and a summary of the text transcription based on the subset of key frames and the subset of key sentences.
A system comprising:
- a memory component;
- and
- a processing device coupled to the memory component, the processing device to perform operations comprising:
- receiving a video input, a text query, and a text transcription of the video input, wherein the video input includes a plurality of frames, and the text transcription includes a plurality of sentences;
- generating a segment of a plurality of segments, the segment comprising a video embedding of a video frame of the plurality of frames, a text embedding of a sentence of the plurality of sentences, and a query embedding of the text query, wherein the video embedding and the text embedding comprise positional information associated with the video frame and the sentence respectively, and wherein a number of video embeddings of the segment is based on a duration between a start time and an end time associated with the sentence of the text embedding;
- determining, by a multimodal summarization model that inputs the plurality of segments, a subset of key frames of the plurality of frames dependent on the text query and a subset of key sentences of the plurality of sentences dependent on the text query;
- and
- providing a query-focused summary of the video input and a query-focused summary of the text transcription based on the subset of key frames and the subset of key sentences.
Dependencies are inferred from the claim text.