Method and system for accelerated morphological analysis
AssigneeXerox Corporation
InventorAndreas Eisele
Life of this patent
Estimating the term…
Abstract
A system provides accelerated morphological analysis and in particular a speed-up of morphological look-up via a caching mechanism. The system determines whether each incoming token in a token stream is unique or recurring. Unique tokens, which occur for the first time in the token stream, are marked with a unique numerical identification (ID). A pointer is added to recurring tokens, which already occurred in the token stream, and directed towards the unique numerical ID which was defined for the respective token when occurring for the first time. A morphological look-up is performed on the unique tokens. Subsequently, the tokens carrying the pointer are detected and replaced with the results of morphological look-up stored under the unique numerical ID of the respective unique token.
Claims20
A method for performing morphology analysis of a natural language document, comprising the steps of:
- inputting the natural language document as an input text,
- tokenize the input text, thereby producing a token stream of a plurality of tokens,
- comparing each token of the token stream to a list of previously encountered tokens in the token stream,
- determining whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is a unique token in the token stream without corresponding to the at least one previously encountered token in the list,
- adding the unique token to the list of previously encountered tokens, marking the unique token with an identification,
- adding a pointer directed towards the identification for the recurring token,
- performing a morphological look-up only on the unique token, thereby producing a morphological look-up result,
- storing said morphological look-up result for the unique token together with the identification,
- reading the morphological look-up result for the recurring token,
- joining the morphological look-up results for the unique and recurring tokens, thereby producing a stream of morphological analyses, and
- outputting the stream of morphological analyses.
A system for performing morphology analysis of a natural language document, comprising:
- a tokenizer for tokenizing an input document, thereby producing a token stream of a plurality of tokens,
- a pre-processor that compares each token of the token stream to a list of previously encountered tokens in the token stream, determines whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is a unique token in the token stream without corresponding to the at least one previously encountered token in the list, adds the unique token to the list of previously encountered tokens, marks the unique token with an identification and adds a pointer directed towards the identification for the recurring token,
- a morphological look-up module for performing a morphological look-up only on the unique token, thereby producing a morphological look-up result,
- memory for storing said morphological look-up result for the unique token together with the identification,
- a post-processor that replaces a token that carries the pointer with the morphological look-up result stored in the memory under the respective identification.
A system for performing morphology analysis of a natural language document, comprising:
- a tokenizer for tokenizing an input document, thereby producing a token stream of a plurality of tokens,
- a morphological look-up module for producing results of a morphological look-up,
- memory for storing the results of the morphological look-up for a unique token of the plurality of tokens together with an identification,
- a control unit for controlling the morphological look-up module and the memory;
- wherein the control unit:
- compares each token of the token stream to a list of previously encountered tokens in the token stream, determines whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is the unique token in the token stream without corresponding to the at least one previously encountered token in the list,
- adds the unique token to the list of previously encountered tokens,
- marks the unique token with the identification,
- adds a pointer directed towards the identification for the recurring token,
- initializes the morphological look-up for the unique token, and
- replaces the recurring token marked with the pointer by the results of the morphological look-up stored in the memory under the respective identification.
Dependencies are inferred from the claim text.