Systems and methods for enabling similarity search using loss-permitted candidates representation

A similarity search system, comprising:

• at least one processor;
• and
• a memory operatively connected to the at least one processor, wherein the at least one processor is configured to:
• generate a reverse index from entity data stored in an entity datastore, wherein the reverse index maps characteristic values to lists of entity identifiers and occurrence counts;
• apply a compaction process to reduce storage requirements by removing entity identifier lists for characteristic values having occurrence counts above a threshold;
• store the compacted reverse index in a database;
• receive similarity search queries for target entities;
• retrieve reverse index entries corresponding to characteristic values of the target entities;
• and
• calculate similarity scores between the target entities and candidate entities using the occurrence counts to determine the discriminative power of shared characteristic values.

A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• creating a reverse index structure from a corpus of entities, each entity having an identifier and characteristic values, wherein the reverse index structure comprises entries mapping each unique characteristic value to a list of entity identifiers containing that characteristic value and a count of entities containing that characteristic value;
• compacting the reverse index structure by eliminating entity identifier lists for characteristic values where the count exceeds a frequency threshold, thereby reducing storage requirements while preserving count information;
• and
• executing similarity search operations by accessing the compacted reverse index structure to retrieve count information for characteristic values of a query entity and using the count information to weight the significance of matching characteristic values during similarity calculations.