Enhanced ai-based audio-visual processing
AssigneeNVIDIA Corporation
InventorsMansata KAMAL, Vishesh Gupta, Rohit Singh
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure relate to enhanced AI-based audio-visual processing. Various aspects integrate multimodal analysis functionality that seamlessly combines and/or selects from audio, video, and/or text data to provide a holistic understanding of multimedia content. Relative to existing technologies, such an approach enables more accurate and contextually aware interpretations by leveraging the full spectrum of available information in multimedia content. By integrating these disparate sources of data, various embodiments achieve a more nuanced analysis that captures the complexity and richness of real-world multimedia scenarios.
Claims20
One or more processors comprising processing circuitry to:
- receive a user query associated with a video;
- access a first embedding representing audio data of the video;
- access a second embedding representing image data of the video;
- and
- generate, based at least on a model processing a user query and a combination of a representation of the first embedding and a representation of the second embedding, a response to the user query.
A system comprising one or more processors to:
- decode audio data into an audio stream and video data into a video stream;
- generate a natural language transcript from the audio stream, the natural language transcript associated with corresponding timestamps;
- based at least on analyzing a frame of the video stream, generate visual metadata associated with the video stream;
- and
- generate a natural language response based at least on at least one of:
- aligning the frame with a respective timestamp of the corresponding timestamps;
- or
- combining or selecting from a representation of at least a portion of the natural language transcript and a representation of at least a portion of the visual metadata.
A method comprising:
- receiving, using a computing device, audio data of a video and image data of the video;
- receiving, using the computing device, a user query;
- sending, using one or more application programming interfaces (APIs) of a container executed using the computing device, and to one or more models hosted using a remote computing device, data representative of the user query and at least one of the audio data or the image data;
- and
- receiving, using the one or more APIs and from the remote computing device responsive to the sending of the data, a natural language response to the user query, the natural language response generated using the one or more models;
- and
- causing presentation, at least one of visually or audibly, of the natural language response using the computing device.
Dependencies are inferred from the claim text.