Video captioning generation system and method
AssigneeLemon Inc.
InventorsLinjie Yang, Heng Wang, Yuhan Shen, Longyin Wen, Haichao Yu
Life of this patent
Estimating the term…
Abstract
A system and a method are provided that include a processor executing a caption generation program to receive an input video, sample video frames from the input video, extract video frames from the input video, extract video embeddings and audio embeddings from the video frames, including local video tokens and local audio tokens, respectively, input the local video tokens and the local audio tokens into at least a transformer layer of a cross-modal encoder to generate multi-modal embeddings, and generate video captions based on the multi-modal embeddings using a caption decoder.
Claims18
A video captioning generation system comprising:
- a processor and memory of a computing device, the processor being configured to execute a program using portions of memory to:
- receive an input video;
- extract video frames from the input video;
- extract video embeddings and audio embeddings from the video frames, the video embeddings including local video tokens and global video tokens, and the audio embeddings including local audio tokens and global audio tokens;
- input the local video tokens and the local audio tokens into at least a transformer layer of a cross-modal encoder to generate multi-modal embeddings;
- input the global video tokens, the global audio tokens, the local video tokens, and the local audio tokens into a global cross fusion module of the cross-modal encoder to generate fused local video tokens, fused global video tokens, fused local audio tokens, and fused global audio tokens;
- and
- generate video captions based on the multi-modal embeddings using a caption decoder, the multi-modal embeddings including the fused local video tokens, the fused global video tokens, the fused local audio tokens, and the fused global audio tokens.
A video captioning generation method comprising:
- receiving an input video;
- extracting video frames from the input video;
- extracting video embeddings and audio embeddings from the video frames, the video embeddings including local video tokens and global video tokens, and the audio embeddings including local audio tokens and global audio tokens, respectively;
- inputting the local video tokens and the local audio tokens into at least a transformer layer of a cross-modal encoder to generate multi-modal embeddings;
- performing global cross fusion on the global video tokens, the global audio tokens, the local video tokens, and the local audio token to generate fused local video tokens, fused global video tokens, fused local audio tokens, and fused global audio tokens;
- and
- generating video captions based on the multi-modal embeddings using a caption decoder, the multi-modal embeddings including the fused local video tokens, the fused global video tokens, the fused local audio tokens, and the fused global audio tokens.
A video captioning generation system comprising:
- a processor and memory of a computing device, the processor being configured to execute a program using portions of memory to:
- receive an input video;
- extract video frames from the input video;
- extract video embeddings including local video tokens and global video tokens;
- extract audio embeddings comprising local audio tokens and global audio tokens corresponding to the local video tokens and the global video tokens;
- input the local video tokens and the local audio tokens into at least a transformer layer of a cross-modal encoder to generate multi-modal embeddings;
- input the global video tokens, the global audio tokens, the local video tokens, and the local audio tokens into a global cross fusion module of the cross-modal encoder to generate fused local video tokens, fused global video tokens, fused local audio tokens, and fused global audio tokens;
- and
- generate video captions based on the multi-modal embeddings using a caption decoder, the multi-modal embeddings including the fused local video tokens, the fused global video tokens, the fused local audio tokens, and the fused global audio tokens.
Dependencies are inferred from the claim text.