Optimizing large language models with domain-oriented model compression

A system for optimizing large language models (LLM) with domain-oriented model compression, comprising:

• a memory device;
• and
• one or more processor devices operatively coupled with the memory device to:
• determine importance weights for general knowledge in a trained LLM, pretrained with deep learning, by computing an error when removing a weight from the trained LLM;
• optimize the trained LLM iteratively to obtain a domain-compressed LLM with domain knowledge while maintaining general knowledge by further performing steps to:
• fine-tune the trained LLM with domain knowledge using the importance weights for general knowledge to obtain a fine-tuned LLM;
• determine importance weights for domain knowledge in the LLM with a regularization term by using gradient descent to optimize parameters when the fine-tuned LLM is trained with domain knowledge;
• prune learned knowledge based on importance weights for domain knowledge;
• and
• perform corrective action on a monitored entity using the domain compressed LLM.

A non-transitory computer program product comprising a computer-readable storage medium including program code for optimizing large language models (LLM) with domain-oriented model compression, wherein the program code when executed on a computer causes the computer to:

• determine importance weights for general knowledge in a trained LLM, pretrained with deep learning, by computing an error when removing a weight from the trained LLM;
• optimize the trained LLM iteratively to obtain a domain-compressed LLM with domain knowledge while maintaining general knowledge by further performing steps to:
• fine-tune the trained LLM with domain knowledge using the importance weights for general knowledge to obtain a fine-tuned LLM;
• determine importance weights for domain knowledge in the LLM with a regularization term by using gradient descent to optimize parameters when the fine-tuned LLM is trained with domain knowledge;
• prune learned knowledge based on importance weights for domain knowledge by removing weights that have importance scores lower than a sparsity threshold;
• and
• perform corrective action on a monitored entity using the domain compressed LLM.