Visual chain-of-thought reasoning for multimodal language models

A method of eliciting Chain-of-Thought reasoning from a multimodal Large Language model (MLLM) of a multimodal assistant system, the method comprising:

• receiving a multimodal input at a user interface component of the multimodal assistant system, the multimodal input including an input image and a natural language task description which describes a structured task to perform based on the input image;
• generating a visual Chain-of-Thought (v-CoT) prompt for the MLLM of the multimodal assistant system using a prompt generating component, the prompt generating component generating the v-CoT prompt by combining the input image, the natural language task description, and a v-CoT instruction, the v-CoT instruction defining a series of steps for the MLLM to take in generating an output that solves the task conditioned on the input image and the natural language task description and that utilizes CoT reasoning, the series of steps including:
• instructing the MLLM to describe relevant information derived from the image and relevant image artifacts required to generate the output;
• instructing the MLLM to reason over the relevant information and relevant image artifacts to determine a solution for the task;
• and
• including the solution in the output;
• providing the v-CoT prompt to the MLLM as an input;
• receiving the output generated by the MLLM based on the v-CoT prompt;
• and
• presenting the output on the user interface for the multimodal assistant system.

A non-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to perform functions of:

• receiving a multimodal input at a user interface component of a multimodal assistant system, the multimodal input including an input image and a natural language task description which describes a structured task to perform based on the input image;
• generating a visual Chain-of-Thought (v-CoT) prompt for a multimodal Large Language Model (MLLM) of the multimodal assistant system using a prompt generating component that is configured to generate the v-CoT prompt by combining the input image, the natural language task description, and a v-CoT instruction, the v-CoT instruction defining a series of steps for the MLLM to take in generating an output that solves the task conditioned on the input image and the natural language task description and that utilizes CoT reasoning, the series of steps including:
• instructing the MLLM to describe relevant information derived from the image and relevant image artifacts required to generate the output;
• instructing the MLLM to reason over the relevant information and relevant image artifacts to determine a solution for the task;
• and
• including the solution in the output;
• feeding the v-CoT prompt to the MLLM as input;
• receiving the output generated by the MLLM based on the v-CoT prompt;
• and
• presenting the output on the user interface for the multimodal assistant system.