Transfer learning system for automated software engineering tasks
AssigneeMicrosoft Technology Licensing, LLC
InventorsColin Bruce Clement, Dawn Drain, Neelakantan Sundaresan, Alexey Svyatkovskiy
Life of this patent
Estimating the term…
Abstract
A transfer learning system is used for the development of neural transformer models pertaining to software engineering tasks. The transfer learning system trains source code domain neural transformer models with attention in various configurations on a large corpus of unsupervised training dataset of source code programs and/or source code-related natural language text. A web service provides the trained models for use in developing a model that may be fine-tuned on a supervised training dataset associated with a software engineering task thereby generating a tool to perform the software engineering task.
Claims20
A web service, comprising:
- a storage server including a custom training dataset;
- a first computing device having a first processor and a first memory device, wherein the first memory device includes a plurality of source code domain neural transformer models with attention, the plurality of source code domain neural transformer models with attention including configurations of an encoder-only neural transformer with attention, a decoder-only neural transformer with attention and/or an encoder-decoder neural transformer with attention, wherein each of the plurality of source code domain neural transformer models with attention associated with a select software engineering task;
- wherein the first memory device includes a program including instructions that perform actions to:
- receive a request for a custom neural transformer model with attention to perform a requested software engineering task;
- obtain the custom training dataset;
- select one of the plurality of source code domain neural transformer models with attention associated with the requested software engineering task;
- and
- fine-tune the select one of the plurality of source code domain neural transformer models with attention with the custom training dataset to generate the custom neural transformer model with attention.
A computer-implemented method, comprising:
- providing a plurality of source code domain neural transformer models with attention, wherein each of the plurality of source code domain neural transformer models includes a memory size and a model size, wherein the model size indicates a number of transformer blocks;
- receiving, through a network, a request for a custom neural transformer model with attention, wherein the request includes a configuration of transformer blocks of the custom neural transformer model with attention;
- and
- when none of the plurality of source code domain neural transformer models with attention meets the requested configuration of transformer blocks:
- selecting one of the plurality of source code domain neural transformer models with attention associated with a larger model size closest to the requested configuration of transformer blocks;
- eliminating select ones of the transformer blocks of the selected one of the plurality of source code domain neural transformer models with attention;
- and
- fine-tuning the select one of the plurality of source code domain neural transformer models with attention with a custom training dataset to generate the custom neural transformer model with attention.
A computer-implemented method, comprising:
- accessing a plurality of pre-trained source code domain neural transformer models with attention, the plurality of pre-trained source code domain neural transformer models with attention including configurations of an encoder-only neural transformer with attention, a decoder-only neural transformer with attention and/or an encoder-decoder neural transformer with attention, each of the plurality of pre-trained source code domain neural transformer models with attention associated with a software engineering task;
- obtaining a supervised training dataset for a specific software engineering task;
- selecting one of the plurality of pre-trained source code domain neural transformer models with attention associated with the specific software engineering task;
- and
- fine-tuning the selected one of the plurality of pre-trained source code domain neural transformer models with attention with the supervised training dataset to perform the specific software engineering task.
Dependencies are inferred from the claim text.