System and method for out-of-distribution detection of visual content using general representation and informative content

A system comprising:

• one or more hardware computing devices configured to:
• define a set of text embeddings from a set of hierarchical text prompts using a text encoder, the set of hierarchical text prompts define a primary informative prompt and a secondary informative prompt associated with the primary informative prompt,
• define a plurality of feature representations of an input image using an image encoder;
• change a value of one or more identified feature representations among the plurality of feature representations to mask the one or more identified feature representation and define a general feature representation of the input image based on a class-specific threshold indicative of boundary between a class-specific feature and a general feature;
• and
• classify the input image based on an out-of-distribution (OOD) score determined using a similarity analysis of the general feature representation and the set of text embeddings.

A non-transitory computer-readable medium comprising instructions for a multi-modal out-of-distribution detection system that, when executed by one or more hardware computing devices cause the one or more hardware computing devices to perform operations including to:

• define a feature representation matrix of a set of in-distribution (ID) images using a first image encoder;
• define a set of mean variances for the feature representation matrix using principal component analysis of the feature representation matrix;
• select a class-specific threshold based on a mean variance diminishing gradient point at which a rate of change of the mean variance is zero;
• define a set of text embeddings from a set of hierarchical text prompts using a text encoder, the set of hierarchical text prompt define a primary informative prompt and a secondary informative prompt associated with the primary informative prompt;
• define a plurality of feature representations of an input image using a second image encoder;
• change a value of one or more identified feature representations among the plurality of feature representations to mask the one or more identified feature representation and define a general feature representation of the input image based on a class-specific threshold indicative of boundary between a class-specific feature and a general feature;
• calculate an out-of-distribution (OOD) score using softmax analysis of the general feature representation and the set of text embeddings;
• and
• classify the input image at ID or OOD based on the OOD.