Extension for lexer algorithms to handle unicode efficiently
AssigneeMicrosoft Corporation
InventorVincenzo Lombardi
Life of this patent
Estimating the term…
Abstract
Lexical groups are created using lexer state transitions associated with a character set. Characters that cause the lexer to transition to the same state, regardless of the current state, are put in the same group. The state transition table is then created with row entries corresponding to lexical groups instead of single characters. The resulting state transition table can be searched much faster, and takes up much less space then the prior art state transition tables. This results in faster and less memory intensive lexer programs.
Claims35
A method for tokenizing an input file, comprising:
- receiving a character from the input file;
- determining a lexical group for the received character;
- determining a next state transition for a lexer using the lexical group and a current state of the lexer;
- and
- outputting a token if the next state transition is to a final state.
A system for tokenizing an input file by a lexer, the system comprising:
- a reading component for reading a character from an input file;
- a buffer component for storing the read character, and previously read characters, if any in a character buffer;
- a lexical group component for generating a lexical group from the read character;
- a state component for determining a next state transition from the lexical group and a current state;
- and
- a token generating component for generating a token from the characters in the character buffer if the next state transition is a final state.
A method for generating lexical groups for a programming language from a set of characters, comprising:
- creating a first lexical group corresponding to the set of characters that are letters;
- and
- identifying characters that are valid in identifiers in the programming language, and creating a second lexical group corresponding to non-letter characters that are valid in identifiers.
A computer-readable medium with computer-executable instructions stored thereon for performing the steps of:
- receiving a character from an input file;
- determining a lexical group for the received character;
- determining a next state transition for the lexer using the lexical group and a current state of the lexer;
- and
- outputting a token if the next state transition is to a final state.
Dependencies are inferred from the claim text.