Leveraging semantic information for a multi-domain visual agent

A system for leveraging semantic information for a multi-domain visual agent, comprising:

• a memory device;
• one or more processor devices operatively coupled with the memory device to:
• sample questions from question templates for domain-specific label spaces to obtain a unified label space;
• map domain-specific labels from the domain-specific label spaces into natural language descriptions (NLD) to obtain mapped NLD;
• generate prompts by combining the questions sampled from the unified label space and the mapped NLD;
• learn the semantic information by iteratively generating outputs from tokens extracted from the prompts using a large-language model (LLM);
• and
• train the multi-domain visual agent (MDVA) using the semantic information to obtain a trained MDVA.

A non-transitory computer program product comprising a computer-readable storage medium including program code for leveraging semantic information for a multi-domain visual agent, wherein the program code when executed on a computer causes the computer to:

• sample questions from question templates for domain-specific label spaces to obtain a unified label space;
• map domain-specific labels from the domain-specific label spaces into natural language descriptions (NLD) to obtain mapped NLD;
• generate prompts by combining the questions sampled from the unified label space and the mapped NLD;
• learn the semantic information by iteratively generating outputs from tokens extracted from the prompts using a large-language model (LLM);
• and
• train the multi-domain visual agent (MDVA) using the semantic information to obtain a trained MDVA.