Dynamic Refinement of Custom Classes Using Zero-Shot Image Classifiers

A system, comprising:

• one or more memories;
• and
• one or more processors configured to execute instructions stored in the one or more memories to:
• capture images of an environment in real-time using an image-capturing device;
• generate image embeddings of the captured images using a trained Contrastive Language-Image Pre-training (CLIP) model;
• receive a text prompt from a user indicating a first object or event;
• generate a text embedding of the text prompt using the CLIP model;
• compute similarity scores between the text embedding and the image embeddings;
• determine a highest similarity score of the similarity scores;
• determine that the highest similarity score exceeds a predefined threshold;
• in response to determining that the highest similarity score exceeds the predefined threshold:
• identify, in the respective captured image that corresponds to the highest similarity score, a second object or event that correlates with the first object or event;
• store the text prompt and the respective captured image as a custom class for future use by the CLIP model;
• provide an indication of the second object or event to the user;
• receive a response from the user based on the indication;
• and
• update the custom class based on the response.

A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations comprising:

• capturing images of an environment in real-time using an image-capturing device;
• generating image embeddings of the captured images using a trained Contrastive Language-Image Pre-training (CLIP) model;
• receiving a text prompt from a user indicating a first object or event;
• generating a text embedding of the text prompt using the CLIP model;
• computing similarity scores between the text embedding and the image embeddings;
• determining a highest similarity score of the similarity scores;
• determining that the highest similarity score exceeds a predefined threshold;
• in response to determining that the highest similarity score exceeds the predefined threshold:
• identifying, in the respective captured image that corresponds to the highest similarity score, a second object or event that correlates with the first object or event;
• storing the text prompt and the respective captured image as a custom class for future use by the CLIP model;
• providing an indication of the second object or event to the user;
• receiving a response from the user based on the indication;
• and
• updating the custom class based on the response.