Systems and methods for using multimodal llms to identify non-compliant user interfaces

A computer system, the computer system comprising:

• at least one processor;
• at least one memory coupled to the at least one processor, the at least one memory storing instructions that, when executed by the processor, cause the processor to:
• detect a trigger condition;
• prompt a large language model to identify an element of a user interface non-compliant with a guideline, wherein prompting includes providing an image showing at least a portion of the user interface to the large language model;
• receive a first output of the large language model;
• identify, based on the first output, the element that is non-compliant with the guideline;
• and
• present, on a display, an indication that the element is non-compliant.

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, causes the at least one processor to:

• detect a trigger condition;
• prompt a large language model to identify an element of a user interface non-compliant with a guideline, wherein prompting includes providing an image showing at least a portion of the user interface to the large language model;
• receive a first output of the large language model;
• identify, based on the first output, the element that is non-compliant with the guideline;
• and
• present, on a display, an indication that the element is non-compliant.