Open-ended audio tracking system via audio foundational and large language models

A computer-implemented method for executing an open-ended audio tracking system, the method comprising:

• providing an audio segment and text-based descriptions to an Audio Foundational Model (AFM), wherein the text-based descriptions correspond to descriptions of sound events to be detected by the AFM;
• executing the AFM to detect a subset of the sound events that are present within the audio segment;
• providing the corresponding subset of text-based descriptions to a Large Language Model (LLM);
• executing the LLM, wherein executing the LLM comprises:
• classifying the audio segment into an acoustic scene category based on the detected subset of the sound events;
• and
• generating additional text-based descriptions that correspond to other descriptions of sound events that pertain to the acoustic scene category;
• and
• providing the additional text-based descriptions to be used in another iteration of executing the open-ended audio tracking system.

A non-transitory, computer-readable medium storing program instructions that, when executed on or across a processor, cause the processor to:

• provide an audio segment and text-based descriptions to an Audio Foundational Model (AFM), wherein the text-based descriptions correspond to descriptions of sound events to be detected by the AFM;
• execute the AFM to detect a subset of the sound events that are present within the audio segment;
• provide the corresponding subset of text-based descriptions to a Large Language Model (LLM);
• execute the LLM, wherein the execution of the LLM comprises:
• classification of the audio segment into an acoustic scene category based on the detected subset of the sound events;
• and
• generation of additional text-based descriptions that correspond to other descriptions of sound events that pertain to the acoustic scene category;
• and
• provide the additional text-based descriptions to be used in another iteration of execution of the AFM with another audio segment.