Back to results
US 2026/0228489 A1Published application
Systems and methods for mitigating catastrophic forgetting in fine-tuned large language models
AssigneeIntuit Inc.
InventorsGuy LEV, Matan VETZLER, Omer WOSNER, Lior Vassertail AZROEL
Life of this patent
Estimating the term…
Abstract
A system and method are provided for mitigating catastrophic forgetting in fine-tuned large language models.
Claims20
A multi-head attention layer of a transformer architecture comprising:
- a plurality of linear layers configured to receive an input, the plurality of linear layers arranged in multiple depths of linear layers;
- a plurality of attention layers, each attention layer configured to receive an output of a depth of linear layers;
- a gate configured to receive the input and calculate a gate vector as a function of the input and a plurality of learnable weights;
- and
- a concatenation layer configured to concatenate the attention layer outputs and apply the gate vector to the concatenated attention layer outputs to generate an output.
2 more elements
A computer-implemented method, performed by at least one processor, comprising:
- providing input data to a linear layer of a multi-head attention layer of a transformer architecture;
- providing the input data to a gate;
- generating, via the gate, a gate vector as a function of the input data and a plurality of learnable weights;
- analyzing the input data via a plurality of attention layers to create an attention output;
- providing the attention output and the gate vector to a concatenation layer;
- concatenating the attention output;
- and
- applying the gate vector to the concatenated attention output, thereby generating an output.
5 more elements
A computing system comprising:
- a processor;
- and
- a non-transitory computer-readable storage device storing computer-executable instructions, the instructions when executed by the processor cause the processor to perform operations comprising:
- providing input data to a linear layer of a multi-head attention layer of a transformer architecture;
- providing the input data to a gate;
- generating, via the gate, a gate vector as a function of the input data and a plurality of learnable weights;
- analyzing the input data via a plurality of attention layers to create an attention output;
- providing the attention output and the gate vector to a concatenation layer;
- concatenating the attention output;
- and
- applying the gate vector to the concatenated attention output, thereby generating an output.
8 more elements
Dependencies are inferred from the claim text.