Back to results
US 2014/0369597 A1Published application
System and method of classifier ranking for incorporation into enhanced machine learning
AssigneeTexifter, LLC
InventorsStuart William Shulman, Mark James Hoy
Life of this patent
Estimating the term…
Abstract
Systems and methods are disclosed for machine classifiers that employ enhanced machine learning. The machine classification may be automated, based on the input of human classifiers, or a combination of both. The selection of human classifiers is determined by a classifier ranking or scoring process. In addition, data generated by the ranking or scoring process can be used to train the machine classifiers to more accurately classify data.
Claims28
A system for enhanced machine learning using coder rankings, the system comprising:
- a first physical data store configured to store a plurality of data sets including a first data set, each of the plurality of data sets including at least one training data set;
- a second physical data store configured to store coder data related to users who perform data coding;
- a classification computer module in electronic communication with the first and second physical data store, the classification computer module configured to:
- store a first set of selected codes corresponding to a set of data items in a training data set of the first data set, the first set of selected codes provided by a first user via a first classifier system;
- store a second set of selected codes corresponding to the set of data items, the second set of selected codes provided by a second user via a second classifier system, wherein the first set of selected codes is different from the second set of selected codes;
- electronically access an adjudicated set of selected codes adjudicated by a third user via an adjudication system;
- and
- a coder ranking module in electronic communication with the first and second physical data store, the module configured to:
- generate a first trust score corresponding to the first user, the first trust score based at least in part on an accuracy determination of the first set of selected codes as compared to the adjudicated set of selected codes;
- generate a second trust score corresponding to the second user, the second trust score based at least in part on an accuracy determination of the second set of selected codes as compared to the adjudicated set of selected codes;
- wherein the accuracy determination of the first set of selected codes indicates a level of accuracy higher than the accuracy determination of the second set of selected codes and the first trust score corresponding to the first user is higher than the second trust score corresponding to the second user;
- and
- store the first trust score and the second trust score in the second physical data store.
10 more elements
A computer-implemented method of enhanced machine learning using coder rankings, the method comprising:
- under control of one or more computing devices configured with specific computer executable instructions,
- storing a first set of selected codes corresponding to a set of data items in a training data set of a first data set, the first set of selected codes provided by a first user via a first classifier system;
- storing a second set of selected codes corresponding to the set of data items, the second set of selected codes provided by a second user via a second classifier system, wherein the first set of selected codes is different from the second set of selected codes;
- electronically accessing an adjudicated set of selected codes adjudicated by a third user via an adjudication system;
- generating a first trust score corresponding to the first user, the first trust score based at least in part on an accuracy determination of the first set of selected codes as compared to the adjudicated set of selected codes;
- generating a second trust score corresponding to the second user, the second trust score based at least in part on an accuracy determination of the second set of selected codes as compared to the adjudicated set of selected codes,
- wherein the accuracy determination of the first set of selected codes indicates a level of accuracy higher than the accuracy determination of the second set of selected codes and the first trust score corresponding to the first user is higher than the second trust score corresponding to the second user;
- and
- storing the first trust score and the second trust score in the second physical data store.
6 more elements
Non-transitory computer storage having stored thereon a computer program that instructs a computer system to at least:
- store a first set of selected codes corresponding to a set of data items in a training data set of a first data set, the first set of selected codes provided by a first user via a first classifier system;
- store a second set of selected codes corresponding to the set of data items, the second set of selected codes provided by a second user via a second classifier system, wherein the first set of selected codes is different from the second set of selected codes;
- electronically accessing an adjudicated set of selected codes adjudicated by a third user via an adjudication system;
- and
- generate a first trust score corresponding to the first user, the first trust score based at least in part on an accuracy determination of the first set of selected codes as compared to the adjudicated set of selected codes;
- generate a second trust score corresponding to the second user, the second trust score based at least in part on an accuracy determination of the second set of selected codes as compared to the adjudicated set of selected codes;
- wherein the accuracy determination of the first set of selected codes indicates a level of accuracy higher than the accuracy determination of the second set of selected codes and the first trust score corresponding to the first user is higher than the second trust score corresponding to the second user;
- and
- store the first trust score and the second trust score in the second physical data store.
6 more elements
Dependencies are inferred from the claim text.