Neural networks with intention layers
AssigneeGDM Holding LLC
InventorMarta Garnelo Abellanas
Life of this patent
Estimating the term…
Abstract
Systems and methods for processing inputs using neural networks with intention layers. Each intention layer includes one or more intention sub-layers that are each configured to: obtain a query matrix, a key matrix, and a value matrix for the intention sub-layer, wherein at least one of the query matrix, the key matrix, and the value matrix are derived from the layer input to the intention layer; determine a key covariance matrix that estimates a covariance of the key matrix; determine an inverse matrix that represents an inverse of the key covariance matrix; and determine a sub-layer output for the intention sub-layer from the inverse matrix, the query matrix, and the value matrix.
Claims26
A system for performing a task comprising processing a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
- a neural network configured to perform the task, the neural network comprising a plurality of layers that comprise one or more intention layers, each intention layer configured to receive a layer input for the intention layer and to process the layer input to generate a layer output for the intention layer, each intention layer comprising a set of one or more intention sub-layers, each intention sub-layer configured to perform operations comprising:
- obtaining a query matrix, a key matrix, and a value matrix for the intention sub-layer, wherein at least one of the query matrix, the key matrix, and the value matrix are derived from the layer input to the intention layer;
- determining a key covariance matrix that estimates a covariance of the key matrix;
- determining an inverse matrix that represents an inverse of the key covariance matrix;
- and
- determining a sub-layer output for the intention sub-layer from the inverse matrix, the query matrix, and the value matrix.
One or more computer storage media storing instructions that when executed by one or more computer cause the one or more computer to implement:
- a neural network configured to perform the task, the neural network comprising a plurality of layers that comprise one or more intention layers, each intention layer configured to receive a layer input for the intention layer and to process the layer input to generate a layer output for the intention layer, each intention layer comprising a set of one or more intention sub-layers, each intention sub-layer configured to perform operations comprising:
- obtaining a query matrix, a key matrix, and a value matrix for the intention sub-layer, wherein at least one of the query matrix, the key matrix, and the value matrix are derived from the layer input to the intention layer;
- determining a key covariance matrix that estimates a covariance of the key matrix;
- determining an inverse matrix that represents an inverse of the key covariance matrix;
- and
- determining a sub-layer output for the intention sub-layer from the inverse matrix, the query matrix, and the value matrix.
A method comprising:
- receiving a network input;
- and
- processing the network input using a neural network to generate a network output for the network input, the neural network comprising a plurality of layers that comprise one or more intention layers, each intention layer configured to receive a layer input for the intention layer and to process the layer input to generate a layer output for the intention layer, each intention layer comprising a set of one or more intention sub-layers, each intention sub-layer configured to perform operations comprising:
- obtaining a query matrix, a key matrix, and a value matrix for the intention sub-layer, wherein at least one of the query matrix, the key matrix, and the value matrix are derived from the layer input to the intention layer;
- determining a key covariance matrix that estimates a covariance of the key matrix;
- determining an inverse matrix that represents an inverse of the key covariance matrix;
- and
- determining a sub-layer output for the intention sub-layer from the inverse matrix, the query matrix, and the value matrix.
Dependencies are inferred from the claim text.