Apparatus and method for generating data useful in indexing and searching

3 The method of claim 2 further comprising:

• identifying new characters or new character patterns for new conventions in textual data expected in the document data;
• and
• modifying the tokenization rules based upon the new identified characters and the new character patterns;
• wherein the tokens are defined by the identified characters, the new identified characters, the identified character patterns, and the new identified character patterns.

A method for query processing comprising:

• receiving a user query;
• retrieving tokenization rules;
• applying the tokenization rules to the user query to generate one or more search tokens, each having one or more concordable characters from the user query;
• and
• searching an index with the one or more search tokens, the index comprising tokens based on application of the tokenization rules to document data.

A method for computerized searching of document data, comprising:

• receiving the document data;
• retrieving tokenization rules for the document data;
• applying the tokenization rules to the document data to generate a plurality of tokens, each having one or more concordable characters from the document data;
• storing the tokens in a data base;
• receiving a user query derived from a query language and comprising at least one reserved character from a set of concordable characters for use in the query language;
• re-retrieving the tokenization rules;
• applying the re-retrieved tokenization rules to the user query to generate one or more search tokens, each having one or more concordable characters from the user query;
• and
• searching the tokens in the data base with the one or more search tokens.

An apparatus comprising a subsystem for computerized processing of source documents, the source processing subsystem comprising:

• a programmed component for receiving the document data;
• a programmed component for retrieving tokenization rules for the document data;
• and
• a programmed component for applying the tokenization rules to the document data to generate a plurality of tokens, each having one or more concordable characters from the document data.

An apparatus comprising a subsystem for computerized processing of search queries, the query processing subsystem comprising:

• a programmed component for receiving a user query;
• a programmed component for retrieving tokenization rules;
• a programmed component for applying the tokenization rules to the user query to generate one or more search tokens, each having one or more concordable characters from the user query;
• and
• a programmed component for searching an index with the one or more search tokens, the index comprising tokens based on application of the tokenization rules to document data.

An apparatus for computerized searching of document data comprising a source processing subsystem and a query processing subsystem, wherein:

• the source processing subsystem comprises:
• a programmed component for receiving the document data;
• a programmed component for retrieving tokenization rules for the document data;
• and
• a programmed component for applying the tokenization rules to the document data to generate a plurality of tokens for storage in a data base, each of the tokens having one or more concordable characters from the document data;
• and
• the query processing subsystem comprises:
• a programmed component for receiving a user query derived from a query language and comprising at least one reserved character from a set of concordable characters for use in the query language;
• a programmed component for re-retrieving the tokenization rules;
• and
• a programmed component for applying the re-retrieved tokenization rules to the user query to generate one or more search tokens for searching the tokens in the data base, each of the search tokens having one or more concordable characters from the user query.

A computer-readable medium carrying a subsystem program for computerized processing of source documents, the subsystem program comprising:

• instructions for receiving the document data;
• instructions for retrieving tokenization rules for the document data;
• and
• instructions for applying the tokenization rules to the document data to generate a plurality of tokens, each having one or more concordable characters from the document data.

A computer-readable medium carrying a subsystem program for computerized processing of search queries, the subsystem program comprising:

• instructions for receiving a user query;
• instructions for retrieving tokenization rules;
• instructions for applying the tokenization rules to the user query to generate one or more search tokens, each having one or more concordable characters from the user query;
• and
• instructions for searching an index with the one or more search tokens, the index comprising tokens based on application of the tokenization rules to document data.

A computer-readable medium carrying a subsystem program for computerized processing of source documents and a subsystem program for computerized processing of search queries, wherein:

• the source processing subsystem program comprises:
• instructions for receiving the document data;
• instructions for retrieving tokenization rules for the document data;
• and
• instructions for applying the tokenization rules to the document data to generate a plurality of tokens for storage in a data base, each of the tokens having one or more concordable characters from the document data;
• and
• the query processing subsystem program comprises:
• instructions for receiving a user query derived from a query language and comprising at least one reserved character from a set of concordable characters for use in the query language;
• instructions for re-retrieving the tokenization rules;
• and
• instructions for applying the re-retrieved tokenization rules to the user query to generate one or more search tokens for searching the tokens in the data base, each of the search tokens having one or more concordable characters from the user query.