US 8,543,382 B2Granted

Method and system for diacritizing arabic language text

AssigneeKing Abdulaziz City for Science and Technology (KACST)

InventorsMohammad Sulaiman Khorsheed, Hussein Khalid Al-Omari

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present invention provides a method and system for diacritizing a text. The method includes analyzing the text to determine whether the text require one or more diacritical marks of a set of diacritical marks. The text includes a plurality of characters associated with an Arabic language. Thereafter, the text is processed for identifying diacritical marks for each character of the plurality of characters. Subsequently, the diacritical marks of the set of diacritical marks are applied on each character of the plurality of characters based on a context associated with the text to obtain a diacritized Arabic text.

Claims12

Dependent claims

A computer-executable method of diacritizing a text, the method comprising:

storing a Hidden Markov Model in computer memory for establishing a probability for associating a given diacritical mark with a given character of the text;
inputting a sequence of individual characters of the text into a computer processor, the computer processor programmed for:
analyzing the text to determine whether the text requires at least one diacritical mark, the text including a plurality of characters associated with an Arabic language;
converting each character to an ASCII code;
feeding each ASCII code in sequence to the Hidden Markov Model;
applying an expectation-maximization process to each ASCII code starting at one end of the sequence;
transitioning from one diacritical mark to another diacritical mark from a set of diacritical marks for each ASCII code;
recording a probability for each diacritical mark when associated with each current ASCII code;
changing a state of the Hidden Markov Model based on each probability over regularly spaced periods of time;
wherein the Hidden Markov Model transitions from state qiat time t to a state qi at time t+1, where t=1, 2, 3, . . . M;
and i, j=1, 2, . . . , N, and where M represents a number of the transitions and N represents a number of the states;
wherein a transition probability aij, representing a probability that diacritical mark qi appears directly after diacritical mark qi, equals an expected number of transitions from state qi to state qi divided by an expected number of transitions from state qi,
finalizing a diacritical mark having the highest probability for the current ASCII code;
processing each character in the sequence of the text, wherein the Hidden Markov Model bases the probability at least in part on the probability of a diacritical mark applied on one or more preceding characters of the sequence and on a context of the text for determining the probability of a diacritical mark on a given character;
generating a sequence of the diacritical marks corresponding to the sequence of characters;
matching the sequence of diacritical marks with the text to obtain the diacritized text;
and
displaying the diacritized text on an output device.

15 more elements

A system for diacritizing a text, the system comprising:

a memory for storing a Hidden Makov Model, a text, and at least one diacritical mark of a set of diacritical marks, the text comprising a plurality of characters associated with an Arabic language;
a processor coupled to the memory, wherein the processor is configured to:
analyze the text to determine whether the text requires at least one diacritical mark of the set of diacritical marks;
convert each character to an ASCII code;
feed each ASCII code in sequence to the Hidden Markov Model;
apply an expectation-maximization process to each ASCII code starting at one end of the sequence;
transition from one diacritical mark to another diacritical mark from the set of diacritical marks for each ASCII code;
record a probability for each diacritical mark when associated with each current ASCII code;
change a state of the Hidden Markov Model based on each probability over regularly spaced periods of time;
wherein the Hidden Markov Model transitions from state qiat time t to a state qiat time t+1, where t=1, 2, 3, . . . M;
and i, j=1, 2, . . . , N, and where M represents a number of the transitions and N represents a number of the states;
wherein a transition probability aij, representing a probability that diacritical mark qjappears directly after diacritical mark qi, equals an expected number of transitions from state qi to state qj divided by an expected number of transitions from state qi,
finalize a diacritical mark having the highest probability for the current ASCII code;
process each character in the sequence of the text, wherein the Hidden Markov Model bases the probability at least in part on the probability of a diacritical mark applied on one or more preceding characters of the sequence and on a context of the text for determining the probability of a diacritical mark on a given character;
generate a sequence of the diacritical marks corresponding to the sequence of characters;
match the sequence of diacritical marks with the text to obtain the diacritized text;
and
a display to present the diacritized text.

15 more elements

Dependencies are inferred from the claim text.