System and method for pruning a set of symbol-based sequences by relaxing an independence assumption of the sequences

A sequence pruning system, comprising:

• a data structure representing a set of sequences, each sequence in the set of sequences including a first symbol and a context of at least one symbol, a subset of the sequences in the set of sequences each being associated with a respective conditional probability that is based on observations of the respective sequence in training data;
• a scoring function computation component which computes a value of a scoring function for each sequence in the set of represented sequences, the scoring function taking into account the conditional probability for the sequence and a probability distribution of each symbol in the sequence if the respective sequence is removed from the set of sequences, the conditional probability for each sequence in the set of sequences that is not in the subset of sequences being computed as a function of the probability of the respective symbol in a back-off context;
• a sequence selecting component which selects a next one of the represented sequences to be removed, the selection being based on the computed scoring function values;
• and
• an update component which updates the scoring function values of remaining ones of the sequences prior to the sequence selecting component selecting another next one of the represented sequences to be removed;
• an output component which outputs a set of remaining sequences;
• and
• a processor which implements the components.

A sequence pruning method, comprising:

• receiving a set of sequences, each received sequence in the set of sequences including at least one symbol and a context, wherein for at least some of the received sequences, the context comprises at least one respective preceding symbol, each of the received sequences in the set of sequences being associated with a respective conditional probability that is based on observations of the sequence in a corpus;
• representing each of the received sequences as a respective node in a tree structure of nodes in which each of the nodes in the tree structure is directly linked to no more than one node in the tree structure that represents a direct ancestor and wherein at least some of the nodes in the tree structure represent back-off sequences in which the context of a linked node in the tree structure is reduced by at least one symbol;
• computing a value of a scoring function for at least some of the sequences in the tree structure based on respective conditional probabilities, a conditional probability for sequences represented in the tree structure that are not in the set of received sequences being computed as a function of a respective symbol in its back-off context;
• selecting one of the represented sequences to be removed, based on the computed scoring function values, the one of the represented sequences being selected from represented sequences for which there are no remaining sequences represented in the tree structure that consist of at least one symbol and the same sequence as the selected sequence, which precedes the at least one symbol;
• updating a value of a scoring function for at least one remaining one of the sequences, whose scoring function value is changed by the removal of the selecting one of the represented sequences;
• after the updating, repeating the selecting of one of the represented sequences from a remaining set of represented sequences;
• outputting a set of the remaining sequences;
• and
• wherein at least one of the representing, computing, selecting, and updating is performed with a processor.