US 4,290,105 A · Method and apparatus for testing membership in a set through hash coding with allowable errors

Abstract

A machine-implemented process, and apparatus, for performing a set membership test on large sets through the technique of binary hash coding with a known allowable expectation of an error. The present invention does not employ content addressable memory; rather, the present invention performs set membership testing by utilizing a hash function, which produces a randomized plurality of simple address locations within a bulk memory, for each item in the set. A testing for membership comprises employing a logical AND operation upon all values of binary indicators at memory locations addressed by hash values of a test item, to determine whether each and every hash value generated for the test item exactly matches with previously loaded indicators at those address locations in the bulk memory, which in the pereferred embodiment is of the CCD type. The present invention is a machine-implemented process, and employs a known algorithm as part of the overall process. The present invention also essentially comprises synergistic interaction of a hardware item called a "hash board", and a bulk memory. The hash board generates a large number of hash addresses for any given item, using minimal computation time. The expected error rate is a function of the vocabulary size and the total number of indicators in the bulk memory. The bulk memory allows a low error rate for large vocabularies without exceeding the statistically ideal loading density of 50 percent. The hashing technique, together with its hardware implementation, allows a black box approach to general set membership testing. Information stored in the bulk memory can be said to be encrypted because the randomizing process makes it impossible to retrieve the original set of items. A preferred embodiment of the machine-implemented process, and apparatus is for on-line spelling checking of each word which appears in daily newspaper production, against a selective 20,000-word vocabulary, previously stored as a randomized set of simple bit addresses.

Claims15

Dependent claims

A process for performing a set membership test by the technique of binary hash coding with allowable errors, comprising the steps of:

a. receiving a source listing of a number of discrete items which define a set, and generating a number of representations for each of said items;
and
b. randomizing each of the number of representations through a hash step in order to yield a plurality of simple and randomly distributed hash address locations for each item of said set regardless of any variation in length for each item comprising the given set, wherein the number of address locations so generated is chosen as a function of the acceptable probability of an erroneous true when testing an item for membership;
and
c. storing a binary indicator (a single binary value, i.e., 0 to 1) at each of the plurality of hash address locations generated for each item of the set within a bulk storage memory device, without the ability to reconstruct the items of the set given the storage locations;
and
d. receiving a new item and examining said new item for set membership by repeating said generating and hashing steps upon said new item to thereby generate an equal number of new item hash address locations, and performing an AND operation upon all of the indicator values at said new item hash address locations to determine if there is a match with the set of previously stored indicators;
and
e. outputing whether there is or is not a match between all the indicators at the new item locations and the previously stored indicators defining the set.

6 more elements

Apparatus for performing a set membership test by the technique of binary hash coding with allowable errors, wherein the set is represented by a plurality of binary indicators at randomly developed address locations within a bulk memory, wherein further there is no ability to reconstruct items of the set given any or all storage locations, said apparatus comprising:

a. means for receiving a source listing of a number of discrete items which define a set and for generating randomized hash address locations which comprises a non-unique representation of said set, said generating means further comprising a hashing means adapted to operate upon each item of the set with a given number of hashing functions to thereby define a randomized, first plurality of address locations;
b. means for storing an indicator at each of said first plurality of hash address locations generated for each item, wherein each such location is further defined by a bit set within a bulk memory;
and
c. means for receiving and examining a test item for set membership by using said hashing means to generate a second plurality of hash address locations by operating upon said test item with said same given number of hash functions;
and
d. AND means for indicating whether all of said second plurality of address locations are found defined within the set of said first plurality of address locations.

3 more elements

Apparatus for testing the spelling of an alpha/numeric word from a selected language, with allowable errors, through a set membership match, wherein a given vocabulary of correctly spelled words has been coded by binary hash coding and stored randomly within a bulk memory, said apparatus comprising:

a. means for generating a first plurality of hash address locations for each of the words of said given vocabulary, and for randomizing the distribution of said locations throughout the available memory space of said bulk binary memory device;
b. said randomizing means comprising table means for generating approximately five randomized initial representations of each word, and outputting each initial representation through a hash board;
c. said hash board comprising means to randomize each received representation approximately four additional times, thereby producing approximately twenty randomized hash codes for each word in said vocabulary;
and,
d. means for inputting said approximately twenty hash code values for each vocabulary word as simple binary indicators into a bulk memory storage device to thereby set indicators at each one of said first plurality of hash address locations;
and,
e. means for receiving a test word sequentially generating a second plurality of approximately twenty hash address locations for said test word, which comprises means to sequential pass the test word to said table means and then to said hash board means;
and,
f. means for accepting said second plurality of hash address locations and for testing the values of those bits in said bulk memory at each of said second plurality of address locations with the logical operator AND, thereby to determine whether the indicators at all test locations are set, with further means for passing a positive indication if there is a match, otherwise passing a negative indication.

6 more elements

Dependencies are inferred from the claim text.

Method and apparatus for testing membership in a set through hash coding with allowable errors

Life of this patent

Method and apparatus for testing membership in a set through hash coding with allowable errors

Life of this patent

Abstract

Claims15