Model output steering using universal images

A computer implemented method for steering an output of a vision-language model (VLM) comprising:

• receiving a multi-modal input to the VLM comprising an image and text;
• perturbing the image to result in a modified image, the perturbing being configured to induce model activations in the VLM corresponding to a desired behavioral change;
• inputting the modified image and the text into the VLM to generate an output, the output corresponding to the desired behavioral change and being generated without modifying internal states of the VLM;
• and
• providing the output to a consuming application or process;
• wherein:
• the perturbing comprises compositing the image with a universal image;
• the universal image is generated by:
• initializing a baseline image;
• and
• iteratively refining the baseline image to minimize a distance between induced activations generated by inputting a distribution of prompts into the VLM and target activations;
• and
• the target activations are identified by:
• accessing a steering vector known to generate the desired behavioral change;
• and
• monitoring activations at one or more layers of the VLM in response to ingestion of the steering vector by the VLM to identify the target activations.

A computer implemented method for steering outputs of a vision-language model (VLM) comprising:

• receiving each of a plurality of multi-modal inputs to the VLM, each input comprising text and a different image;
• perturbing each image to result in a plurality of modified images, the perturbing being configured to induce model activations in the VLM mimicking activations by the VLM responsive to ingestion of a steering vector;
• inputting each of the modified images and the text into the VLM to generate a plurality of outputs, each of the outputs corresponding to an output corresponding to the steering vector and being generated without modifying internal states of the VLM;
• and
• providing the outputs to a consuming application or process;
• wherein:
• the perturbing comprises compositing each image with a universal image;
• the universal image is generated by:
• initializing a baseline image;
• and
• iteratively refining the baseline image to minimize a distance between induced activations generated by inputting a distribution of prompts into the VLM and target activations;
• and
• the target activations are identified by:
• accessing a steering vector known to generate the desired behavioral change;
• and
• monitoring activations at one or more layers of the VLM in response to ingestion of the steering vector by the VLM to identify the target activations.

A computer implemented method for red teaming a vision-language model (VLM) comprising:

• receiving each of a plurality of multi-modal inputs to the VLM, each input comprising text and a different image;
• perturbing each image to result in a plurality of modified images, the perturbing being configured to induce model activations in the VLM mimicking activations by the VLM responsive to ingestion of each of a plurality of different steering vectors;
• inputting each of the modified images and the text into the VLM to generate a plurality of outputs, each of the outputs corresponding to outputs generated by a corresponding one of the plurality of different steering vectors and being generated without modifying internal states of the VLM;
• and
• providing data characterizing the outputs to a consuming application or process, the provided data identifying vulnerabilities in the VLM;
• wherein:
• the perturbing comprises compositing each image with a universal image;
• the universal image is generated by:
• initializing a baseline image;
• and
• iteratively refining the baseline image to minimize a distance between induced activations generated by inputting a distribution of prompts into the VLM and target activations;
• and
• the target activations are identified by:
• accessing a steering vector known to generate the desired behavioral change;
• and
• monitoring activations at one or more layers of the VLM in response to ingestion of the steering vector by the VLM to identify the target activations.