Image Analysis by Prompting of Machine-Learned Models Using Chain of Thought

One or more memory devices storing non-transitory computer-readable instructions executable to cause one or more processors to perform operations for performing image analysis, the operations comprising:

• obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response;
• inputting, to a machine-learned model, the instructive sequence and an operative image processing query comprising image data, wherein the machine-learned model is configured to process the operative image processing query with attention over the instructive sequence;
• and
• generating, using the machine-learned model and responsive to the operative image processing query, an operative image processing response.

A computing system for performing image analysis, the system comprising:

• one or more processors;
• and
• one or more memory devices storing non-transitory computer-readable instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
• obtaining, by a computing system comprising one or more processors, an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response;
• inputting, by the computing system and to a machine-learned model, the instructive sequence and an operative image processing query comprising image data, wherein the machine-learned model is configured to process the operative image processing query with attention over the instructive sequence;
• and
• generating, by the computing system, using the machine-learned model and responsive to the operative image processing query, an operative image processing response.