Noise removal in multibyte text encodings using statistical models
AssigneeInternational Business Machines Corporation
InventorsJeffrey S. McCarley, Wei-Jing Zhu
Life of this patent
Estimating the term…
Abstract
Disclosed is a method of validating a byte sequence having a plurality of states, the method comprising designating one or more noise states from among the plurality of states; generating a most probable state sequence for the byte sequence; utilizing said state sequence to identify all noise in the byte sequence; and localizing said noise in said noise states. Once localized, the noise may be deleted from the byte sequence.
Claims15
A method of validating a byte sequence, the method comprising:
- defining a plurality of states for the byte sequence;
- designating one or more noise states from among the plurality of states;
- generating a most probable state sequence for the byte sequence;
- utilizing said most probable state sequence to identify all noise in the byte sequence;
- and
- localizing said noise in said noise states.
A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for validating a byte sequence, said method comprising:
- defining a plurality of states for the byte sequence;
- designating one or more noise states from among the plurality of states;
- generating a most probable state sequence for the byte sequence;
- utilizing said state sequence to identify all noise in the byte sequence;
- and
- localizing said noise in said noise states.
A method of validating a byte sequence, the method comprising:
- defining a plurality of states for the byte sequence, including at least one ASCII state;
- designating at least one ASCII state as a noise state;
- generating a most probable state sequence for the byte sequence by a method comprising:
- calculating P(X0 . . . Xn|S0 . . . Sn), representing the conditional probabilities of said byte sequence given a state sequence;
- wherein said calculating P(X0 . . . Xn|S0 . . . Sn) comprises assigning a state label Si to each ith byte Xi of the byte sequence so as to maximize the equation:
- P
- (
- X
- 0
-
-
- …
-
-
- X
- N
- ❘
- S
- 0
-
-
- …
-
-
- S
- N
- )
- =
- P
- 0
-
- (
- S
- 0
- )
- [
- ∏
- i
- =
- 1
- N
-
-
- A
- _
- (
- S
- i
- ❘
- S
- i
- -
- 1
- )
- ]
- [
- ∏
- i
- =
- 0
- N
-
-
- B
- _
-
- (
- X
- i
- ❘
- S
- i
- )
- ]
- wherein P0(S0) is the initial distribution of states;
- Ā(Si|Si−1) is a “state-to-state” transition matrix;
- and {overscore (B)}(Xi|Si) is a “byte-from-state” matrix of the probabilities of generating a byte value Xi given a state S1;
- utilizing said state sequence to identify all noise in the byte sequence;
- localizing said noise in said noise states;
- and
- deleting said noise from the byte sequence.
Dependencies are inferred from the claim text.