Hybrid comparison for unicode text strings consisting primarily of ASCII characters

A computing device, comprising:

• one or more processors;
• memory;
• and
• one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for:
• receiving a first text string S1 with Unicode encoding and a second text string S2 with Unicode encoding;
• computing, for the first text string S1, a first string weight according to a weight function ƒ that computes an ASCII prefix ƒA(S1), computes a Unicode weight suffix ƒU(S1), and concatenates the ASCII prefix to the Unicode weight suffix to form the first string weight ƒ(S1)=ƒA(S1)+ƒU(S1), wherein:
• computing the ASCII prefix for the first text string comprises applying bitwise operations to n-byte contiguous blocks of the first text string to determine whether each block contains only ASCII characters, and replacing accented Unicode characters in the first text string with equivalent unaccented ASCII characters when comparison is designated as accent-insensitive, wherein n is a predefined integer greater than or equal to 4;
• and
• computing the Unicode weight suffix comprises, when there is a first block containing a non-replaceable non-ASCII character, performing a character-by-character Unicode weight lookup beginning with the first block containing a non-replaceable non-ASCII character;
• computing, for the second text string S2, a second string weight according to the weight function ƒ;
• and
• determining whether the first text string and the second text string are equal by comparing the first string weight to the second string weight.

A non-transitory computer readable storage medium storing one or more programs configured for execution by a computing device having one or more processors and memory, the one or more programs comprising instructions for:

• receiving a first text string S1 with Unicode encoding and a second text string S2 with Unicode encoding;
• computing, for the first text string S1, a first string weight according to a weight function ƒ that computes an ASCII prefix ƒA(S1), computes a Unicode weight suffix ƒU(S1), and concatenates the ASCII prefix to the Unicode weight suffix to form the first string weight ƒ(S1)=ƒA(S1)+ƒU(S1), wherein:
• computing the ASCII prefix for the first text string comprises applying bitwise operations to n-byte contiguous blocks of the first text string to determine whether each block contains only ASCII characters, and replacing accented Unicode characters in the first text string with equivalent unaccented ASCII characters when comparison is designated as accent-insensitive, wherein n is a predefined integer greater than or equal to 4;
• and
• computing the Unicode weight suffix comprises, when there is a first block containing a non-replaceable non-ASCII character, performing a character-by-character Unicode weight lookup beginning with the first block containing a non-replaceable non-ASCII character;
• computing, for the second text string S2, a second string weight according to the weight function ƒ;
• and
• determining whether the first text string and the second text string are equal by comparing the first string weight to the second string weight.