Automatic labeler assignment using a model built from multi-labeler data
AssigneeSiemens Medical Solutions USA, Inc.
InventorsGlenn Fung, Romer E. Rosales, Jennifer G. Dy, Yan Yan
Life of this patent
Estimating the term…
Abstract
A method, including receiving multi-labeler data that includes data points labeled by a plurality of labelers; building a model from the multi-labeler data, wherein the model includes an input variable that corresponds to the data points, a label variable that corresponds to true labels for the data points, and variables for the labels given by the labelers; and executing the model, in response to receiving new data points, to determine a level of expertise of the labelers for the new data points.
Claims17
A method, comprising:
- receiving multi-labeler data that includes data points labeled by a plurality of labelers;
- building a model from the multi-labeler data, wherein the model includes an input variable that corresponds to the data points, a label variable that corresponds to true labels for the data points, and variables for the labels given by the labelers, wherein the variables for the labels given by the labelers for a particular data point depend on the input variable that corresponds to that data point and the label variable that corresponds to the true label for that data point;
- and
- executing the model, in response to receiving new data points, to determine a level of expertise of the labelers for the new data points,
- wherein the method is performed using a processor.
A system, comprising:
- a memory device for storing a program;
- a processor in communication with the memory device, the processor operative with the program to:
- receive multi-labeler data that includes data points labeled by a plurality of labelers;
- build a model from the multi-labeler data, wherein the model includes an input variable x that corresponds to the data points, a label variable z that corresponds to true labels for the data points, and variables for the labels y given by each labeler t;
- and
- execute the model, in response to receiving new data points, to determine a level of expertise of the labelers for the new data points,
- wherein the model is represented by,
- p
-
- (
- Y
- ,
- Z
- ❘
- X
- )
- =
- ∏
- i
-
-
- p
-
- (
- z
- i
- ❘
- x
- i
- )
-
- ∏
- i
-
-
-
- p
-
- (
- y
- i
- (
- t
- )
- ❘
- x
- i
- ,
- z
- i
- )
- ,
- where p(zi, xi) is a classifier for classifying new data points, and p(yi(t)|xi,zi) is an error/accuracy estimator for determining labeler expertise.
A computer program product, comprising:
- a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
- computer readable program code configured to perform the steps of:
- receiving multi-labeler data that includes data points labeled by a plurality of labelers;
- building a model from the multi-labeler data, wherein the model includes an input variable that corresponds to the data points, a label variable that corresponds to true labels for the data points, and variables for the labels given by each labeler, wherein the variables for the labels given by the labelers for a particular data point depend on the input variable that corresponds to that data point and the label variable that corresponds to the true label for that data point;
- receiving new data points;
- and
- classifying the new data points using the model.
Dependencies are inferred from the claim text.