Method and computer device for training a large language model
AssigneeGarena Online Private Limited
InventorQian LIU
Life of this patent
Estimating the term…
Abstract
Various embodiments concern a method of training a large language model (LLM) for an unseen task T′ comprising: obtaining a plurality of pre-trained Low-Rank Adaptation (LoRA) modules, using a set of examples Q to obtain a set of weights for the plurality of pre-trained LoRA modules, using the set of weights on the plurality of pre-trained LoRA modules to obtain a fused LoRA module, and applying the fused LoRA module to the LLM to get an adapted LLM for the unseen task T′. Systems and machine-readable media are also provided.
Claims20
A method of training a large language model (LLM) for an unseen task T′ comprising:
- obtaining a plurality of pre-trained Low-Rank Adaptation (LoRA) modules;
- using a set of examples Q to obtain a set of weights for the plurality of pre-trained LoRA modules;
- using the set of weights on the plurality of pre-trained LoRA modules to obtain a fused LoRA module;
- and
- applying the fused LoRA module to the LLM to get an adapted LLM for the unseen task T′.
A computer device for training a large language model (LLM) for an unseen task T′ comprising:
- a processor, a memory, the memory storing at least one program code, the at least one program code loaded and executed by the processor to:
- obtain a plurality of pre-trained Low-Rank Adaptation (LoRA) modules;
- use a set of examples Q to obtain a set of weights for the plurality of pre-trained LoRA modules;
- use the set of weights on the plurality of pre-trained LoRA modules to obtain a fused LoRA module;
- and
- apply the fused LoRA module to the LLM to get an adapted LLM for the unseen task T′.
A computer readable storage medium, characterized in that the storage medium stores at least one program code for execution by a processor to implement operations for:
- obtaining a plurality of pre-trained Low-Rank Adaptation (LoRA) modules;
- using a set of examples Q to obtain a set of weights for the plurality of pre-trained LoRA modules;
- using the set of weights on the plurality of pre-trained LoRA modules to obtain a fused LoRA module;
- and
- applying the fused LoRA module to a large language model (LLM) to get an adapted LLM for an unseen task T′.
Dependencies are inferred from the claim text.