Back to results
US 12,608,594 B2Granted
Machine-learned attention models featuring omnidirectional processing
AssigneeGoogle Inc.
InventorsYi Tay, Da-Cheng Juan, Dara Bahri, Donald Arthur Metzler, Jr., Jai Prakash Gupta, Mostafa Dehghani, Phillip Pham, Vamsi Krishna Aribandi, Zhen Qin
Life of this patent
Estimating the term…
Abstract
Provided are machine-learned attention models that feature omnidirectional processing, example implementations of which can be referred to as Omnidirectional Representations from Transformers (OMNINET). In example models described in the present disclosure, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in some or all of the other tokens across the entire network.
Claims19
A computing system for omnidirectional attention-based processing, the computing system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store:
- a machine-learned multi-layer attention model that comprises a plurality of self-attention layers;
- a machine-learned omnidirectional model configured to:
- receive a plurality of intermediate layer representations respectively generated at two or more of the plurality of self-attention layers of the machine-learned multi-layer attention model;
- attend between a first intermediate layer representation from a first layer of the machine-learned multi-layer attention model and a second intermediate layer representation from a second layer of the machine-learned multi-layer attention model;
- and
- attend between the first intermediate layer representation and a third intermediate layer representation from a third layer of the machine-learned multi-layer attention model;
- and
- instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
- providing a first input to the machine-learned multi-layer attention model to cause generation of a first output;
- providing the plurality of intermediate layer representations respectively generated at the two or more of the plurality of self-attention layers of the machine-learned multi-layer attention model as input to the machine-learned omnidirectional model to cause generation of a second output.
10 more elements
One or more non-transitory computer-readable media that collectively store:
- a multi-layer attention model that comprises a plurality of self-attention layers;
- a omnidirectional model configured to:
- receive a plurality of intermediate layer representations respectively generated at two or more of the plurality of self-attention layers of the multi-layer attention model when processing the first input to generate the first output;
- attend between a first intermediate layer representation from a first layer of the machine-learned multi-layer attention model and a second intermediate layer representation from a second layer of the multi-layer attention model;
- and
- attend between the first intermediate layer representation and a third intermediate layer representation from a third layer of the multi-layer attention model;
- and
- instructions that, when executed, cause a computing system to perform operations, the operations comprising:
- providing a first input to the multi-layer attention model to cause generation of a first output;
- providing the plurality of intermediate layer representations respectively generated at the two or more of the plurality of self-attention layers of the multi-layer attention model as input to the machine-learned omnidirectional model to cause generation of a second output;
- and
- training at least the omnidirectional model based at least in part on a loss value from a-loss function, the loss value based at least in part on the second output.
9 more elements
A computing system for omnidirectional attention-based processing, the computing system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store:
- a machine-learned omnidirectional model configured to:
- receive a plurality of intermediate layer representations respectively generated by two or more of a plurality of self-attention layers of a machine-learned multi-layer attention model;
- attend between a first intermediate layer representation from a first layer of the machine-learned multi-layer attention model and a second intermediate layer representation from a second layer of the multi-layer attention model;
- and
- attend between the first intermediate layer representation and a third intermediate layer representation from a third layer of the multi-layer attention model;
- and
- instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
- providing a first input to the machine-learned multi-layer attention model to cause generation of a first output;
- providing the plurality of intermediate layer representations respectively generated at the two or more of the plurality of self-attention layers of the machine-learned multi-layer attention model as input to the machine-learned omnidirectional model to cause generation of a second output.
9 more elements
Dependencies are inferred from the claim text.