Learnable image masking for securing large language model chatbots

A system comprising:

• a computer processor;
• a network interface for receiving a first new large language model (LLM) prompt comprising a first image portion showing an image and a first text portion showing text;
• an image embedding model executing on the computer processor for causing the computer processor to process the first image portion of the first new LLM prompt to generate a first image embedding prior to transmitting to the LLM;
• a text embedding model executing on the computer processor for causing the computer processor to process the first text portion of the first new LLM prompt to generate a first text embedding prior to transmitting to the LLM;
• an image predictor model executing on the computer processor for causing the computer processor to execute, prior to transmitting to the LLM, the first image embedding and the first text embedding to generate a first image score indicating that the image is redundant of the text;
• masking software executing on the computer processor for causing the computer processor to mask the first image portion of the first new LLM prompt responsive to the first image score failing to satisfy an image threshold to generate a masked prompt;
• an LLM firewall for transmitting the masked prompt, instead of the first new LLM prompt, to an LLM to obtain a first result;
• and
• a user interface for presenting the first result.

A method comprising:

• obtaining a training large language model (LLM) prompt;
• populating a scoring prompt template to score a first response from processing the training LLM prompt across a plurality of factors to obtain an image inclusive prompt;
• populating the scoring prompt template to score a second response from processing the training LLM prompt with an image portion masked across the plurality of factors to obtain an image exclusive prompt;
• generating, with a multimodal LLM, a score using the image inclusive prompt and the image exclusive prompt;
• and
• training the image predictor model with a training dataset comprising the training LLM prompt associated with the score.