US 8,027,938 B1 · Discriminative training in machine learning

Abstract

Systems, methods, and apparatuses including computer program products for machine learning are provided. A method is provided that includes distributing a parameterized model to each worker of a hierarchy of workers, the parameterized model including a plurality of feature functions and corresponding model parameters, processing a portion of training data at each worker of the plurality of workers according to the parameterized model to calculate updates to model parameters, for each worker at a lowest level of the hierarchy of workers, sending the calculated updates to a next higher level worker, for each other worker in the hierarchy of workers, combining updates of the respective worker with updates received from one or more lower level workers, collecting all updates from the workers at a master to generate real updates to the model parameters, and generating an updated model using the real updates to the model parameters.

Claims39

Dependent claims

A method comprising:

distributing a parameterized model to each worker of a plurality of workers, the plurality of workers being arranged in a hierarchy of workers, the parameterized model including a plurality of feature functions representing a plurality of features and corresponding model parameters;
processing a portion of training data at each worker of the plurality of workers according to the parameterized model, the training data including a number of training examples, to calculate updates to model parameters;
for each worker at a lowest level of the hierarchy of workers, sending the calculated updates to a next higher level worker;
for each other worker in the hierarchy of workers, combining updates of the respective worker with updates received from one or more lower level workers;
collecting all updates from the workers at a master to generate real updates to the model parameters;
generating an updated model using the real updates to the model parameters;
and
distributing the updated model to each worker when performing a next iteration.

5 more elements

A method comprising:

receiving training data, the training data including source and target language pairs;
distributing portions of the training data to each worker of a plurality of workers where the plurality of workers are arranged in a hierarchy of workers;
identifying a parameterized model that includes a plurality of features and corresponding parameters, the features including lexical n-gram features;
training the model, including:
distributing the model to each worker of the hierarchy of workers,
processing a portion of training data at each worker of the plurality of workers according to the parameterized model to identify updates to the model parameters,
for each worker at a lowest level of the hierarchy of workers, sending the calculated parameter updates to a next higher level worker,
for each other worker in the hierarchy of workers, combining parameter updates of the respective worker with parameter updates received from one or more lower level workers,
collecting all updates from the workers at a master to generate real updates to the model parameters, and
generating an updated model using the real updates to the parameters;
and
distributing the updated model to each worker when performing a next iteration.

9 more elements

A system comprising:

one or more computers configured to perform operations including:
distributing a parameterized model to each worker of a plurality of workers, the plurality of workers being arranged in a hierarchy of workers, the parameterized model including a plurality of feature functions representing a plurality of features and corresponding model parameters;
processing a portion of training data at each worker of the plurality of workers according to the parameterized model, the training data including a number of training examples, to calculate updates to model parameters;
for each worker at a lowest level of the hierarchy of workers, sending the calculated updates to a next higher level worker;
for each other worker in the hierarchy of workers, combining updates of the respective worker with updates received from one or more lower level workers;
collecting all updates from the workers at a master to generate real updates to the model parameters;
generating an updated model using the real updates to the model parameters;
and
distributing the updated model to each worker when performing a next iteration.

6 more elements

A system comprising:

one or more computers configured to perform operations including:
receiving training data, the training data including source and target language pairs;
distributing portions of the training data to each worker of a plurality of workers where the plurality of workers are arranged in a hierarchy of workers;
identifying a parameterized model that includes a plurality of features and corresponding parameters, the features including lexical n-gram features;
training the model, including:
distributing the model to each worker of the hierarchy of workers,
processing a portion of training data at each worker of the plurality of workers according to the parameterized model to identify updates to the model parameters,
for each worker at a lowest level of the hierarchy of workers, sending the calculated parameter updates to a next higher level worker,
for each other worker in the hierarchy of workers, combining parameter updates of the respective worker with parameter updates received from one or more lower level workers,
collecting all updates from the workers at a master to generate real updates to the model parameters, and
generating an updated model using the real updates to the parameters;
and
distributing the updated model to each worker when performing a next iteration.

10 more elements

A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations comprising:

distributing a parameterized model to each worker of a plurality of workers, the plurality of workers being arranged in a hierarchy of workers, the parameterized model including a plurality of feature functions representing a plurality of features and corresponding model parameters;
processing a portion of training data at each worker of the plurality of workers according to the parameterized model, the training data including a number of training examples, to calculate updates to model parameters;
for each worker at a lowest level of the hierarchy of workers, sending the calculated updates to a next higher level worker;
for each other worker in the hierarchy of workers, combining updates of the respective worker with updates received from one or more lower level workers;
collecting all updates from the workers at a master to generate real updates to the model parameters;
generating an updated model using the real updates to the model parameters;
and
distributing the updated model to each worker when performing a next iteration.