Loading patent…
AssigneeHewlett-Packard Company
InventorJeffrey B. Caldwell
Estimating the term…
The present invention provides a system and method for building a lexical analyzer that can scan multibyte character sets. The present invention factors regular expressions that contain multibyte characters, so that a single byte finite state automata can be constructed. In particular, the present invention provides a computer-based system and method for tokenizing a source program written in a programming language that is represented by both single byte values and two byte values. The present invention includes a mechanism for building a lexical analyzer that is configured to accept an input specification. The input specification typically includes a regular expression(s) and a corresponding associated action(s). The present invention also including a mechanism for factoring the regular expression(s), if the regular expression(s) contains at least one two byte character, into a regular expression(s) containing only single byte characters. The present invention will result in both a significantly smaller lexical analyzer which uses substantially less memory and a lexical analyzer which easily recognizes a heterogeneous input stream of single and two byte characters.
A computer-based system for building a lexical analyzer that can recognize a character set that is represented by both single byte values and multibyte values or only multibyte values, comprising:
A computer-based method for building a lexical analyzer that can recognize a character set that has both single byte and multibyte characters or only multibyte characters, comprising the steps of:
A method of factoring a regular expression with a two byte character class range into a regular expression with a single byte character class range, were the character class range is of the form [ay-bz], where a, b, y, and z are single byte values, the method comprising the steps of:
A computer-based system for building a lexical analyzer that can recognize a character set that is represented by both single byte values and multibyte values or only multibyte values, comprising:
Dependencies are inferred from the claim text.