Gated linear networks

A neural network system comprising:

• one or more computers, and
• one or more storage devices on which are stored instructions, that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for implementing one or more gated linear networks,
• wherein each gated linear network corresponds to a respective data value in an output data sample and is configured to generate a network probability output that defines a probability distribution over possible values for the respective data value,
• wherein each gated linear network comprises a plurality of layers arranged in a hierarchy of layers,
• wherein the plurality of layers comprises a plurality of gated linear layers,
• wherein each gated linear layer has one or more nodes, and
• wherein the operations comprise, for each node in each gated linear layer:
• receiving a plurality of inputs from nodes in a layer below the gated linear layer in the hierarchy of layers;
• receiving side information for the node;
• combining the plurality of inputs according to a set of weights defined by the side information to generate an initial output;
• generating, from the initial output, a node probability output that defines a probability distribution over possible values for the corresponding data value;
• and
• providing as output the node probability output for the corresponding data value.

A method performed by a neural network system implementing one or more gated linear networks,

• wherein each gated linear network corresponds to a respective data value in an output data sample and is configured to generate a network probability output that defines a probability distribution over possible values for the corresponding data value,
• wherein each gated linear network comprises a plurality of layers arranged in a hierarchy of layers,
• wherein the plurality of layers comprises a plurality of gated linear layers,
• wherein each gated linear layer has one or more nodes, and
• wherein the method comprises, for each node in each gated linear layer:
• receiving a plurality of inputs from nodes in a layer below the gated linear layer in the hierarchy of layers;
• receiving side information for the node;
• combining the plurality of inputs according to a set of weights defined by the side information to generate an initial output;
• generating, from the initial output, a node probability output that defines a probability distribution over possible values for the corresponding data value;
• and
• providing as output the node probability output for the corresponding data value.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, causes the one or more computers to perform operations for implementing one or more gated linear networks,

• wherein each gated linear network corresponds to a respective data value in an output data sample and is configured to generate a network probability output that defines a probability distribution over possible values for the corresponding data value,
• wherein each gated linear network comprises a plurality of layers arranged in a hierarchy of layers,
• wherein the plurality of layers comprises a plurality of gated linear layers,
• wherein each gated linear layer has one or more nodes, and
• wherein the operations comprise, for each node in each gated linear layer:
• receiving a plurality of inputs from nodes in a layer below the gated linear layer in the hierarchy of layers;
• receiving side information for the node;
• combining the plurality of inputs according to a set of weights defined by the side information to generate an initial output;
• generating, from the initial output, a node probability output that defines a probability distribution over possible values for the corresponding data value;
• and
• providing as output the node probability output for the corresponding data value.