US 2017/0364810 A1 · METHOD OF GENERATING A TRAINING OBJECT FOR TRAINING A MACHINE LEARNING ALGORITHM

Abstract

There is disclosed a computer implemented method of generating a training object for training a machine learning algorithm (MLA). The method comprises: acquiring a digital training document to be used in the training; transmitting the digital training document to a plurality of assessors, transmitting further including indicating a range of possible labels for the assessors to assess from, the range of possible labels including at least a first possible label and a second possible label; obtaining from each of the plurality of assessors a selected label to form a pool of selected labels; generating a consensus label distribution based on the pool of selected labels, the consensus label distribution representing a range of perceived labels for the digital training document and an associated probability score for each of the perceived labels; and training the machine learning algorithm using the digital training document and the consensus label distribution.

Claims20

Dependent claims

A computer implemented method of generating a training object for training a machine learning algorithm, the training object including a digital training document and an assigned label, the method executable at a training server, the method comprising:

acquiring the digital training document to be used in the training;
transmitting, via a communication network, the digital training document to a plurality of assessors, transmitting further including indicating a range of possible labels for the assessors to assess from the range of possible labels including at least a first possible label and a second possible label;
obtaining from each of the plurality of assessors a selected label to form a pool of selected labels;
generating a consensus label distribution based on the pool of selected labels, the consensus label distribution representing a range of perceived labels for the digital training document and an associated probability score for each of the perceived labels;
and
training the machine learning algorithm using the digital training document and the consensus label distribution.

3 more elements

A training server for training a ranking application, the ranking application for ranking search results, the training server comprising:

a network interface for communicatively coupling to a communication network;
a processor coupled to the network interface, the processor configured to:
acquire the digital training document to be used in the training;
transmit, via the communication network, the digital training document to a plurality of assessors, transmitting further including indicating a range of possible labels for the assessors to assess from, the range of possible labels including at least a first possible label and a second possible label;
obtain from each of the plurality of assessors a selected label to form a pool of selected labels;
generate a consensus label distribution based on the pool of selected labels, the consensus label distribution representing a range of perceived labels for the digital training document and an associated probability score for each of the perceived labels;
and
train the machine learning algorithm using the digital training document and the consensus label distribution.

5 more elements

Dependencies are inferred from the claim text.

Method of generating a training object for training a machine learning algorithm

Life of this patent

Method of generating a training object for training a machine learning algorithm

Life of this patent

Abstract

Claims20