Automatic algorithm generation

A computer based pattern recognition construction system comprising:

• a feature module having:
• a feature selection module arranged to derive a candidate feature set having a plurality of candidate features;
• and,
• a feature extract module arranged to interact with a plurality of digitally stored data objects to extract feature vectors therefrom, wherein said feature vectors are derived from said candidate feature set;
• a classifier training module having:
• a classifier selection module arranged to select a candidate classifier set having at least one candidate classifier;
• and,
• a training module arranged to train said candidate classifier set based upon said feature vectors generated by said feature extract module;
• a classifier effectiveness module arranged to evaluate said candidate classifier set and generate at least one performance measure;
• a first feedback path from said classifier effectiveness module to said feature module;
• and,
• a second feedback path from said classifier effectiveness module to said classifier training module, wherein said at least one performance measure generated by said classifier effectiveness module determines whether feedback is required to said feature module via said first feedback path to modify said feature vectors, to said classifier training module via said second feedback path to modify said candidate classifier set, or to both.

A pattern recognition construction system comprising:

• a feature module arranged to interact with a plurality of pre-classified training data objects and generate training feature vectors therefrom, said training feature vectors defined by a candidate feature set comprising a plurality of candidate features;
• a training module arranged to select and train at least one candidate classifier based upon said training feature vectors generated by said feature module;
• a feature extract module arranged to interact with a plurality of pre-classified testing data objects and generate testing feature vectors therefrom, said testing feature vectors defined by said candidate feature set;
• a classifier module arranged to classify said testing feature vectors using said at least one candidate classifier, and,
• an effectiveness module arranged to determine at least one performance measure for each candidate classifier trained by said training module, or used by said classifier module, said at least one performance measure arranged to enable refinement of said at least one classifier through iterative feedback from said effectiveness module to at least one of said feature module to modify said training feature vectors, and said training module to modify said at least one candidate classifier, until a predetermined stopping criterion is met.

A pattern recognition construction system comprising:

• a feature module comprising:
• a feature selection module arranged to generate a candidate feature set comprising a plurality of candidate features;
• and,
• a first feature extract module arranged to extract training feature vectors from a pre-classified training data set based upon said candidate feature set;
• a training module comprising:
• a classifier selection module arranged to select a classifier set comprising at least one candidate classifier defined by a classifier algorithm;
• and,
• a classifier training module arranged to train said at least one candidate classifier based upon said training feature vectors;
• a second feature extract module arranged to extract testing feature vectors from a pre-classified testing data set based upon said candidate feature set;
• a first classifier module arranged to classify said testing feature vectors using said at least one candidate classifier;
• a classifier effectiveness module arranged to evaluate said candidate classifier set either trained by said classifier training module, or used by said first classifier module to classify said testing feature vectors, and generate at least one performance measure;
• a first feedback path from said classifier effectiveness module to said feature module, wherein said feature module is arranged to add new candidate features, remove select ones of said plurality of candidate features, modify select ones of said plurality of candidate features, and extract additional feature vectors from said plurality of data objects in any combination to modify said feature vectors;
• and,
• a second feedback path from said classifier effectiveness module to said training is module, wherein said training module is arranged to add a candidate classifier, remove a select one of said at least one candidate classifier, retrain said at least one candidate classifier based upon modified parameters of said at least one classifier, and retrain said at least one candidate classifier based upon modified feature vectors in any combination to modify said at least one candidate classifier, wherein said feedback repeats iteratively until a predetermined stopping criterion is met, said candidate feature set at the time said stopping criterion is met defining a final feature set, and a select one of said at least one candidate classifier defining a final classifier.

A pattern recognition construction system comprising:

• at least one processor;
• a storage device;
• an output device;
• and,
• software executable by said at least one processor for:
• accessing in said storage device digitally stored representations of data objects;
• generating a candidate feature set having a plurality of candidate features;
• extracting feature vectors from said digitally stored representations of data objects based upon said candidate feature set;
• selecting at least one candidate classifier defining a candidate classifier set;
• training said at least one candidate classifier using said feature vectors;
• and,
• iteratively refining said at least one classifier until a predetermined stopping criterion is met, said at least one classifier refined by:
• providing a performance measure for each of said at least one candidate classifier;
• and, performing at least one of:
• extracting additional feature vectors and training said at least one candidate classifier thereon;
• modifying said candidate feature set, wherein feature vectors are extracted from said digitally stored representations of data objects based upon the modified candidate feature set, and said at least one candidate classifier is retrained thereon;
• modifying said candidate feature set by either adding at least one new candidate classifier or removing at least one candidate classifier from said candidate classifier set, wherein said classifier set is retrained on said feature vectors;
• and,
• modifying at least one parameter of at least one candidate classifier, wherein said candidate classifier is retrained, wherein said output device is adapted to output at least one candidate classifier in said classifier set and said candidate feature set after said predetermined stopping criterion is met.

A pattern recognition construction system comprising:

• a storage device;
• an output device;
• and,
• a processor programmed to:
• access from said storage device, digitally stored representations of data objects;
• extract feature vectors from said digitally stored representations of data objects based upon a candidate feature set;
• and,
• train a classifier set comprising at least one candidate classifier using said feature vectors;
• provide a performance measure for each of said at least one candidate classifier;
• and,
• refine said classifier set based upon said performance measure by at least one of a modification to said candidate feature set and a modification to said candidate feature set.

A pattern recognition construction system comprising:

• a feature module arranged to interact with a plurality of data objects and generate feature vectors therefrom, said feature vectors defined by a candidate feature set comprising a plurality of candidate features;
• a training module arranged to select and train at least one candidate classifier based upon said feature vectors generated by said feature module;
• and,
• an effectiveness module arranged to determine at least one performance measure for each candidate classifier and enable refinement thereof, wherein:
• at least one of said feature module and said training module are arranged to accept feedback of said performance measure from said effectiveness module;
• said feature module, where arranged to accept said feedback, is further arranged to modify said feature vectors in response to feedback of said performance measure to said feature module;
• said training module, where arranged to accept said feedback, is further arranged to modify said candidate classifier in response to feedback of said performance measure to said training module.

A computer readable carrier including a computer program that causes a computer to automate the development of classifiers, the computer program configured to cause said computer to perform operations comprising:

• accessing a data set comprising a plurality of data objects;
• identifying a candidate feature set based upon at least one candidate feature;
• using a feature extraction process to extract feature vectors from said data set based upon said candidate feature set;
• using a training process to train at least one candidate classifier from said feature vectors;
• using an effectiveness process to provide a performance measure of said at least one candidate classifier;
• and,
• iteratively refining said candidate classifier based upon said performance measure until a predetermined stopping criterion is met by performing for each iteration, at least one of:
• extracting additional feature vectors from said data set based upon said candidate feature set, wherein said candidate classifier is trained by said training process using said additional feature vectors and a new performance measure of said candidate classifier is recomputed by said effectiveness process;
• modifying said candidate feature set, wherein said feature extraction process extracts new feature vectors from said data objects based upon the modified version of said candidate feature set, said candidate classifier is retrained using said new feature vectors and a new performance measure of said candidate classifier is recomputed;
• and,
• modifying said candidate classifier, wherein the modified version of said candidate classifier is retrained using said feature vectors, and a new performance measure is recomputed.

A pattern recognition construction system comprising:

• means for integrating into a feedback driven system that can iterate until a predetermined stopping criterion is met having:
• means for extracting feature vectors from a training set of data objects based upon a candidate feature set;
• means for training at least one candidate classifier based upon said feature set;
• means for providing a performance measure of said at least one candidate classifier;
• and,
• means for refining said at least one candidate classifier by at least one of modifying said feature vectors and modifying said at least one classifier;
• and,
• means for outputting at least one candidate classifier.

A computer automated method for pattern recognition construction comprising:

• identifying a candidate feature set based upon at least one candidate feature;
• executing a feature extraction process computer code to extract feature vectors from a training set of digitally stored representations of data objects based upon said candidate feature set;
• executing a training process computer code to train a candidate classifier set having at least one candidate classifier on said feature vectors;
• executing an effectiveness process computer code to provide a performance measure of said at least one candidate classifier;
• and,
• iteratively developing said candidate classifier set based upon said performance measure until a predetermined stopping criterion is met by performing for each iteration, at least one of:
• executing said feature extraction process computer code to extract additional feature vectors from said training set based upon said candidate feature set;
• modifying said candidate feature set, wherein said feature extraction process computer code is executed to extract new feature vectors from said data objects based upon the modified version of said candidate feature set;
• modifying said candidate classifier set;
• retraining said candidate classifier set;
• and,
• providing a new performance measure of said at least one candidate classifier, wherein a final feature set is defined by the candidate feature set at the time said predetermined stopping criterion is met, and a final classifier is defined by a select one of said at least one candidate classifier when said predetermined stopping criterion is met.

A method for automating pattern recognition comprising:

• accessing a data set comprising a plurality of data objects;
• identifying a candidate feature set based upon at least one candidate feature;
• using a feature extraction process to extract feature vectors from said data set based upon said candidate feature set;
• using a training process to train a candidate classifier from said feature vectors;
• using an effectiveness process to provide a performance measure of said at least one candidate classifier;
• and,
• iteratively refining said candidate classifier based upon said performance measure until a predetermined stopping criterion is met by performing for each iteration, at least one of:
• extracting additional feature vectors from said data set based upon said candidate feature set, wherein said candidate classifier is trained by said training process using said additional feature vectors and a new performance measure of said candidate classifier is recomputed by said effectiveness process;
• modifying said candidate feature set, wherein said feature extraction process extracts new feature vectors from said data objects based upon the modified version of said candidate feature set, said candidate classifier is retrained using said new feature vectors and a new performance measure of said candidate classifier is recomputed;
• and,
• modifying said candidate classifier, wherein the modified version of said candidate classifier is retrained using said feature vectors, and a new performance measure is recomputed.

A method of performing automated pattern recognition comprising:

• integrating into a computer environment:
• a feature selection module arranged select features to define a feature set;
• a feature extraction module arranged to extract feature vectors from data objects based upon said feature set;
• a classifier selection module arranged to select at least one classifier;
• a classifier training module arranged to train said at least one classifier selected by said classifier selection module based upon feature vectors extracted from said feature extraction module;
• and,
• a classifier performance evaluation module arranged to report at least one performance measure for each classifier trained by said classifier training module;
• providing a training data set comprising a plurality of digitally stored representations of pre-classified data objects;
• using said feature selection module to define a candidate feature set;
• using said feature extraction module to extract training feature vectors from said training data set based upon said candidate feature set;
• using said classifier selection module to select at least one candidate classifier;
• using said classifier training module to train said at least one candidate classifier using said training feature vectors extracted by said feature extraction module;
• using said classifier performance evaluation module to report at least one performance measure for each candidate classifier;
• and,
• using said report of said at least one performance measure to direct change to at least one of said training feature vectors and said at least one candidate classifier.

A method of refining a classifier comprising:

• obtaining a data set;
• sampling from said data set, a training set of data, and an evaluation set of data;
• developing a plurality of candidate classifiers using said training data;
• evaluating said plurality of candidate classifiers using said evaluation data;
• performing a first bootstrap operation to determine the performance of each of said candidate classifiers;
• performing a second bootstrap operation to determine the performance of each of said candidate classifiers;
• examining a bias evident in the results of said second bootstrap;
• applying a bias correction to the first bootstrap results based upon said bias in said second bootstrap;
• obtaining at least one of an estimate and a confidence interval of the bias corrected performance of each of said plurality of candidate classifiers to derive at least one performance measure;
• and,
• using said at least one performance measure as feedback to improve at least one of said plurality of candidate classifiers.