Similarity sharding

A system comprising:

• at least one programmable processor;
• and
• a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
• parsing a datatype item into one or more tokens;
• extracting at least one selected token from the parsed datatype item, the at least one selected token comprising a character string including one or more characters;
• standardizing the character string of the at least one selected token;
• extracting a first character from the one or more characters included in the at least one standardized selected token;
• and
• assigning the datatype item to a select shard of a plurality of shards via character distribution lookup based on the extracted first character, wherein the character distribution look up comprises a pre-computed distribution of characters based on analysis of first character frequencies of a plurality of representative geographic locale-specific data samples, the pre-computed distribution of characters configured to balance frequency of occurrence of datatype items associated with the geographic locale across the plurality of shards.

A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

• computing first character frequency statistical data for a geographic locale from a plurality of representative locale-specific data samples associated with the geographic locale;
• based at least on the first character frequency statistical data for the geographic locale, generating a character distribution lookup that is configured to predictively balance frequency of occurrence of datatype items associated with the geographic locale across a plurality of shards;
• parsing a datatype item into one or more tokens;
• extracting at least one selected token from the parsed datatype item, the at least one selected token comprising a character string including one or more characters;
• standardizing the character string of the at least one selected token;
• extracting a first character from the one or more characters included in the at least one standardized selected token;
• and
• assigning, based on the extracted first character, the datatype item to a select shard of the plurality of shards via the character distribution lookup.