Embedding constrained and unconstrained optimization programs as neural network layers

A computing device configured to embed a convex optimization program as an optimization layer in a neural network architecture comprising a plurality of layers, the computing device comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the computing device to:
• determine a set of decision variables as parameters associated with the convex optimization program, wherein a coefficient matrix corresponding to one or more constraints of the convex optimization program imposes constraints on the decision variables;
• determine a set of intermediary functions, wherein one function of the set of intermediary functions is equal to a cost term associated with a range of permitted values and a second function of the set of intermediary functions is equal to infinity for a range of unpermitted values;
• generate, based on applying a scattering coordinate transform to the decision variables, a set of network variables associated with the optimization layer, wherein the set of network variables comprises an input value of the optimization layer and an intermediate value of the optimization layer;
• generate, based on the coefficient matrix, a linear component of the optimization layer;
• generate, based on a second transformation to the set of intermediary functions, a non-linear component of the optimization layer;
• generate, based on an input of the optimization layer, the optimization layer as a direct form structure;
• receive, by the optimization layer and from a prior layer of the neural network architecture, input values;
• iteratively compute, based on the linear and non-linear components of the optimization layer and the input values and intermediate values of the optimization layer, fixed point values of the network variables;
• determine, based on applying an inverse of the scattering coordinate transform to the fixed point values of the network variables, fixed point values for the decision variables;
• provide, by the optimization layer and based on the determined fixed point values for the decision variables, a first output to a next layer of the neural network architecture;
• provide, based on the first output of the neural network architecture, a second output of the neural network architecture;
• and
• generate, by the neural network architecture and based on a trained set of parameters associated with the convex optimization program, one or more predictions.

A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause a computing device to perform steps configured to embed a convex optimization program as an optimization layer in a neural network architecture comprising a plurality of layers, the steps comprising:

• determining decision variables as a set of parameters associated with the convex optimization program, wherein a coefficient matrix corresponding to one or more constraints of the convex optimization program imposes linear constraints on the decision variables;
• determining a set of intermediary functions equal to a cost term associated with a range of permitted values and equal to infinity for a range of unpermitted values, wherein the one or more constraints of the convex optimization program define the range of permitted values and the range of unpermitted values;
• generating, based the decision variables, a set of network variables associated with the optimization layer;
• generating, based on the coefficient matrix, a linear component of the optimization layer;
• generating, based on the set of intermediary functions, a non-linear component of the optimization layer;
• generating, based on an input of the optimization layer, the optimization layer as a residual form structure;
• receiving, by the optimization layer and from a prior layer of the neural network architecture, input values;
• iteratively computing, based on the linear and non-linear components of the optimization layer and the input values of the optimization layer, fixed point values of the network variables;
• determining, based on the fixed point values of the network variables, fixed point values for the decision variables;
• providing, by the optimization layer and based on the determined fixed point values for the decision variables, a first output to a next layer of the neural network architecture;
• providing, by the optimization layer, a second output, wherein the second output is based on the first output;
• determining, based on second output, an error;
• backpropagating the determined error through the plurality of layers;
• and
• generating, by the neural network architecture and based on a trained set of parameters associated with the convex optimization program, one or more predictions.