# US 12,001,961 B2 · Embedding constrained and unconstrained optimization programs as neural network layers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,961 B2 - Document ID: US12001961B2 - Status: Granted - Kind: B2 - Application number: 18/079,588 - Filed: 2022-12-12 - Published: 2023-04-06 - Granted: 2024-06-04 - Earliest claimed: 2019-02-15 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Tarek Aziz Lahlou; Christopher Larson; Oluwatobi Olabiyi - Web page: https://patentgrep.com/patent/US12001961B2 ## CPC - G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks - G06F 17/11: for solving equations {, e.g. nonlinear equations, general mathematical optimization problems (optimization specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0499: Feedforward networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/045: Combinations of networks - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Aspects discussed herein may relate to methods and techniques for embedding constrained and unconstrained optimization programs as layers in a neural network architecture. Systems are provided that implement a method of solving a particular optimization problem by a neural network architecture. Prior systems required use of external software to pre-solve optimization programs so that previously determined parameters could be used as fixed input in the neural network architecture. Aspects described herein may transform the structure of common optimization problems/programs into forms suitable for use in a neural network. This transformation may be invertible, allowing the system to learn the solution to the optimization program using gradient descent techniques via backpropagation of errors through the neural network architecture. Thus these optimization layers may be solved via operation of the neural network itself. ## Claims ### Claim 1 A computer-implemented method for embedding a convex optimization program as an optimization layer in a neural network architecture comprising a plurality of layers, the method comprising: determining a set of parameters associated with the convex optimization program comprising associated primal decision variables and dual decision variables, wherein a coefficient matrix corresponding to one or more constraints of the convex optimization program imposes linear feasibility constraints on the decision variables; determining a set of intermediary functions, wherein one function of the set of intermediary functions is equal to a cost term associated with a range of permitted values and a second function of the set of intermediary functions is equal to infinity for a range of unpermitted values; generating, based on applying a scattering coordinate transform to the decision variables, a set of network variables associated with the optimization layer, wherein the set of network variables comprises an input value of the optimization layer and an intermediate value of the optimization layer; generating, based on the coefficient matrix, a linear component of the optimization layer; generating, based on a second transformation to the set of intermediary functions, a non-linear component of the optimization layer, wherein the second transformation is based on the intermediate value of the optimization layer; iteratively determining, based on the linear and non-linear components of the optimization layer and the input values and intermediate values of the optimization layer, fixed point values of the network variables; determining, based on applying an inverse of the scattering coordinate transform to the fixed point values of the network variables, fixed point values for the decision variables; providing, by the optimization layer and based on the determined fixed point values for the decision variables, a first output to a next layer of the neural network architecture; providing, by the optimization layer, a second output, wherein the second output is based on the first output; determining, based on the second output, an error; backpropagating the determined error through the plurality of layers; and generating, by the neural network architecture and based on a trained set of parameters associated with the convex optimization program, one or more predictions. ### Claim 2 (depends on 1) The method of claim 1, wherein the backpropagating comprises determining an updated set of parameters associated with the convex optimization program based on applying gradient descent to the linear component and the non-linear component of the optimization layer. ### Claim 3 (depends on 1) The method of claim 1, wherein the one or more constraints of the convex optimization program define the range of permitted values and the range of unpermitted values. ### Claim 4 (depends on 1) The method of claim 1, wherein the neural network architecture is configured to generate predictions regarding speech recognition. ### Claim 5 (depends on 1) The method of claim 1, wherein the neural network architecture is configured to generate predictions regarding image recognition. ### Claim 6 (depends on 1) The method of claim 1, wherein the neural network architecture comprises a convolutional neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein the neural network architecture comprises at least one of a recurrent neural network or a feed forward neural network. ### Claim 8 (depends on 1) The method of claim 1, wherein the optimization layer has a direct form structure, from an input of the optimization layer, consisting of a single path comprising a single matrix multiplication followed by a non-linearity. ### Claim 9 (depends on 1) The method of claim 1, wherein the optimization layer has a residual form structure, from an input of the optimization layer, comprising: a first path, comprising a single matrix multiplication followed by a non-linearity, a second path, comprising an identity path from the input of the optimization layer, and a weighted combination of the input and a product of the single matrix multiplication and the non-linearity. ### Claim 10 (depends on 1) The method of claim 1, wherein the convex optimization program is one of a linear optimization program or a quadratic optimization program. ### Claim 11 A computing device configured to embed a convex optimization program as an optimization layer in a neural network architecture comprising a plurality of layers, the computing device comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the computing device to: determine a set of decision variables as parameters associated with the convex optimization program, wherein a coefficient matrix corresponding to one or more constraints of the convex optimization program imposes constraints on the decision variables; determine a set of intermediary functions, wherein one function of the set of intermediary functions is equal to a cost term associated with a range of permitted values and a second function of the set of intermediary functions is equal to infinity for a range of unpermitted values; generate, based on applying a scattering coordinate transform to the decision variables, a set of network variables associated with the optimization layer, wherein the set of network variables comprises an input value of the optimization layer and an intermediate value of the optimization layer; generate, based on the coefficient matrix, a linear component of the optimization layer; generate, based on a second transformation to the set of intermediary functions, a non-linear component of the optimization layer; generate, based on an input of the optimization layer, the optimization layer as a direct form structure; receive, by the optimization layer and from a prior layer of the neural network architecture, input values; iteratively compute, based on the linear and non-linear components of the optimization layer and the input values and intermediate values of the optimization layer, fixed point values of the network variables; determine, based on applying an inverse of the scattering coordinate transform to the fixed point values of the network variables, fixed point values for the decision variables; provide, by the optimization layer and based on the determined fixed point values for the decision variables, a first output to a next layer of the neural network architecture; provide, based on the first output of the neural network architecture, a second output of the neural network architecture; and generate, by the neural network architecture and based on a trained set of parameters associated with the convex optimization program, one or more predictions. ### Claim 12 (depends on 11) The computing device of claim 11, further comprising: determine, based on the second output of the neural network architecture and before generating the one or more predictions, an error; and backpropagate the determined error through the plurality of layers by at least determining an updated set of parameters associated with the convex optimization program based on applying gradient descent to the linear non-linear component of the optimization layer. ### Claim 13 (depends on 11) The computing device of claim 11, wherein the direct for structure comprises a single path comprising a single matrix multiplication followed by a non-linearity. ### Claim 14 (depends on 11) The computing device of claim 11, wherein the one or more constraints of the convex optimization program define the range of permitted values and the range of unpermitted values of the set of intermediary functions. ### Claim 15 (depends on 11) The computing device of claim 11, wherein the convex optimization program is at least one of a linear optimization program or a quadratic optimization program. ### Claim 16 (depends on 11) The computing device of claim 11, wherein the neural network architecture comprises at least one of a convolutional neural network, a recurrent neural network, or a feed forward neural network. ### Claim 17 A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause a computing device to perform steps configured to embed a convex optimization program as an optimization layer in a neural network architecture comprising a plurality of layers, the steps comprising: determining decision variables as a set of parameters associated with the convex optimization program, wherein a coefficient matrix corresponding to one or more constraints of the convex optimization program imposes linear constraints on the decision variables; determining a set of intermediary functions equal to a cost term associated with a range of permitted values and equal to infinity for a range of unpermitted values, wherein the one or more constraints of the convex optimization program define the range of permitted values and the range of unpermitted values; generating, based the decision variables, a set of network variables associated with the optimization layer; generating, based on the coefficient matrix, a linear component of the optimization layer; generating, based on the set of intermediary functions, a non-linear component of the optimization layer; generating, based on an input of the optimization layer, the optimization layer as a residual form structure; receiving, by the optimization layer and from a prior layer of the neural network architecture, input values; iteratively computing, based on the linear and non-linear components of the optimization layer and the input values of the optimization layer, fixed point values of the network variables; determining, based on the fixed point values of the network variables, fixed point values for the decision variables; providing, by the optimization layer and based on the determined fixed point values for the decision variables, a first output to a next layer of the neural network architecture; providing, by the optimization layer, a second output, wherein the second output is based on the first output; determining, based on second output, an error; backpropagating the determined error through the plurality of layers; and generating, by the neural network architecture and based on a trained set of parameters associated with the convex optimization program, one or more predictions. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the residual form structure wherein the residual form structure comprises: a first path, comprising a single matrix multiplication followed by a non-linearity; a second path, comprising an identity path from the input of the optimization layer; and a weighted combination of the input and a product of the single matrix multiplication and the non-linearity. ### Claim 19 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the backpropagating comprises determining an updated set of parameters associated with the convex optimization program based on applying gradient descent to the linear component and the non-linear component of the optimization layer. ### Claim 20 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the set of network variables comprises an input value of the optimization layer and an intermediate value of the optimization layer.