Multi-agent visual perception

A non-transitory computer readable medium storing code for image processing, the code comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• obtaining an image and a query about the image;
• generating, using an orchestrator vision-language model, an action based on the image and the query;
• generating, using a specialized vision-language model, an analysis result based the image and the action;
• and
• generating, using the orchestrator vision-language model, an answer to the query based on the analysis result.

A system for image processing, comprising:

• a memory component;
• a processing device coupled to the memory component, the processing device configured to perform operations comprising:
• obtaining an image and a query about the image;
• generating, using an orchestrator vision-language model, an action based on the image and the query;
• generating, using a vision expert model, a vision analysis result based the image and the action;
• and
• generating, using the orchestrator vision-language model, an answer to the query based on the vision analysis result.