Method for citation identification

A computer-implemented method for generating a citation database from a corpus of documents stored in a computer readable memory, wherein the corpus of documents includes at least ten million documents, the method comprising for each of the documents in the corpus of documents, by executing instructions stored on a processor readable media of at least one computing device:

• searching, by the at least one computing device, the document for an entity identifier corresponding to an entity and, if an instance of the entity identifier is detected in the document, determining a portion of the document around the instance of the entity identifier as a target text, wherein the entity is associated with a product catalogue, the product catalogue comprising a plurality of product identifiers;
• automatically applying, by the at least one computing device, a first regular expression to the target text, wherein the first regular expression is configured to match one or more of the product identifiers;
• identifying, by the at least one computing device, one or more tokens from the target text that match the first regular expression;
• and
• determining, by the at least one computing device, if any of the one or more tokens corresponds to a product identifier from the product catalogue and, if so, which product identifier;
• if a product identifier from the product catalogue is determined to be cited in the target text, automatically adding, by the at least one computing device, an entry to a citation database linking the document and the product identifier;
• wherein determining if any of the one or more tokens corresponds to a product identifier comprises an iterative process, each iteration of the iterative process including:
• determining a set of prefixes for the plurality of product identifiers, each prefix having a predetermined number of characters;
• generating a set of prefix regular expressions corresponding to the set of prefixes;
• applying the prefix regular expressions to the one or more tokens, and keeping only the prefix regular expressions that return a match;
• wherein for each subsequent iteration, the predetermined number of characters is increased by one, and the set of prefixes is determined to include only prefixes that match a prefix regular expression kept in the previous iteration;
• and
• wherein, a token is determined to correspond to a product identifier if a prefix regular expression matches the token, and if the predetermined number of characters corresponds to a number of characters in the product identifier.