System and method for knowledge-based audio-text modeling via automatic multimodal graph construction

A system for knowledge-based audio-text modeling via automatic multimodal graph construction, comprising:

• one or more hardware computing devices configured to:
• receive an audio dataset, the audio dataset including clips of audio data, wherein each of the clips of the audio data is paired with corresponding metadata descriptive of the audio contents of the respective clip;
• identify graph nodes of interest from a semantic network, the graph nodes being descriptive of semantics of a knowledge domain of the audio dataset;
• utilize a large language model (LLM) for categorizing the metadata into the graph nodes and for inferring supplemental data for the graph nodes for which there is no metadata, producing an extracted knowledge graph;
• validate the extracted knowledge graph utilizing the LLM to perform relation verification of edges between the graph nodes of the extracted knowledge graph, thereby mitigating hallucination effects in the categorizing and inferring of the supplemental data;
• and
• utilize the knowledge graph, as validated, for a downstream application, including providing the knowledge graph to an audio generation model as a prompt to generate audio data based on the knowledge graph.

A non-transitory computer-readable medium comprising instructions for a knowledge-based audio-text modeling via automatic multimodal graph construction that, when executed by one or more hardware computing devices cause the one or more hardware computing devices to perform operations including to:

• receive an audio dataset, the audio dataset including clips of audio data, wherein each of the clips of the audio data is paired with corresponding metadata descriptive of the audio contents of the respective clip;
• identify graph nodes of interest from a semantic network, the graph nodes being descriptive of semantics of a knowledge domain of the audio dataset;
• utilize a large language model (LLM) for categorizing the metadata into the graph nodes and for inferring supplemental data for the graph nodes for which there is no metadata, producing an extracted knowledge graph;
• validate the extracted knowledge graph utilizing the LLM to perform relation verification of edges between the graph nodes of the extracted knowledge graph, thereby mitigating hallucination effects in the categorizing and inferring of the supplemental data;
• and
• utilize the knowledge graph, as validated, for a downstream application, including providing the knowledge graph to an audio generation model as a prompt to generate audio data based on the knowledge graph.