US 2018/0121807 A1Published application

When output units must obey hard constraints

AssigneeOracle International Corporation

InventorsMichael Wick, Jean-Baptiste Tristan, Jay Yoon Lee

Life of this patent

Estimating the term…

Loading patent…

Abstract

Embodiments employ an inference method for neural networks that enforces deterministic constraints on outputs without performing post-processing or expensive discrete search over the feasible space. Instead, for each input, the continuous weights are nudged until the network's unconstrained inference procedure generates an output that satisfies the constraints. This is achieved by expressing the hard constraints as an optimization problem over the continuous weights and employing backpropagation to change the weights of the network. Embodiments optimize over the energy of the violating outputs; since the weights directly determine the output through the energy, embodiments are able to manipulate the unconstrained inference procedure to produce outputs that conform to global constraints.

Claims20

Dependent claims

A method comprising:

training a neural network, during a training phase of the neural network, to produce a trained neural network;
after completing the training phase of the neural network:
performing inference, using the trained neural network comprising a first set of weights, on a particular unlabeled input to produce a first output;
determining whether the first output violates one or more constraints on output values of the trained neural network;
in response to determining that the first output violates the one or more constraints on output values of the trained neural network, using backpropagation to adjust the first set of weights of the trained neural network to produce an adjusted trained neural network comprising a second set of weights;
wherein the first set of weights is different than the second set of weights;
performing inference, using the adjusted trained neural network, on the particular unlabeled input to produce a second output;
wherein the second output is different from the first output;
and
wherein the method is performed by one or more computing devices.

7 more elements

One or more non-transitory computer-readable media storing one or more sequences of instructions which, when executed by one or more processors, cause:

training a neural network, during a training phase of the neural network, to produce a trained neural network;
after completing the training phase of the neural network:
performing inference, using the trained neural network comprising a first set of weights, on a particular unlabeled input to produce a first output;
determining whether the first output violates one or more constraints on output values of the trained neural network;
in response to determining that the first output violates the one or more constraints on output values of the trained neural network, using backpropagation to adjust the first set of weights of the trained neural network to produce an adjusted trained neural network comprising a second set of weights;
wherein the first set of weights is different than the second set of weights;
performing inference, using the adjusted trained neural network, on the particular unlabeled input to produce a second output;
wherein the second output is different from the first output.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/450,933Filed 6 Mar 2017
Priority: 31 Oct 2016
Published: 3 May 2018
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G06N 3/084
Learning methods: backpropagation, e.g. using gradient descent
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks
G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
G06N 3/0455
Combinations of networks: auto-encoder networks; Encoder-decoder networks