# US 12,008,459 B2 · Multi-task machine learning architectures and training procedures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,459 B2 - Document ID: US12008459B2 - Status: Granted - Kind: B2 - Application number: 16/443,440 - Filed: 2019-06-17 - Published: 2020-10-22 - Granted: 2024-06-11 - Earliest claimed: 2019-04-19 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Weizhu Chen; Pengcheng He; Xiaodong Liu; Jianfeng Gao - Web page: https://patentgrep.com/patent/US12008459B2 ## CPC - G06N 3/047: Probabilistic or stochastic networks - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06F 40/216: using statistical methods - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06N 3/048: Activation functions - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks. The method can also involve performing a tuning stage on the one or more shared layers and the two or more task-specific layers using respective task-specific objectives. ## Claims ### Claim 1 A method performed on a computing device, the method comprising: evaluating a plurality of candidate multi-task teacher instances of a multi-task machine learning model, each candidate multi-task teacher instance having one or more shared layers, a first task-specific layer that performs a first task, and a second task-specific layer that performs a second task; based at least on the evaluating, selecting one or more first multi-task teacher instances of the plurality of candidate multi-task teacher instances for the first task and one or more second multi-task teacher instances of the plurality of candidate multi-task teacher instances for the second task; and training a multi-task student instance of the multi-task machine learning model, the multi-task student instance having the one or more shared layers, the first task-specific layer, and the second task-specific layer, the training comprising using first outputs of the one or more first multi-task teacher instances to train the first task-specific layer of the multi-task student instance and using second outputs of the one or more second multi-task teacher instances to train the second task-specific layer of the multi-task student instance. ### Claim 2 (depends on 1) The method of claim 1, further comprising: training the multi-task student instance using an objective function defined based at least on the first outputs and the second outputs. ### Claim 3 (depends on 2) The method of claim 2, the objective function being further defined based at least on labels associated with labeled data processed to obtain the first outputs and the second outputs. ### Claim 4 (depends on 1) The method of claim 1, wherein the selecting comprises: training the plurality of candidate multi-task teacher instances to perform the first task and the second task; selecting the one or more first multi-task teacher instances based at least on accuracy of the one or more first multi-task teacher instances at performing the first task; and selecting the one or more second multi-task teacher instances based at least on accuracy of the one or more second multi-task teacher instances at performing the second task. ### Claim 5 (depends on 4) The method of claim 4, wherein the training comprises: averaging first outputs of multiple first multi-task teacher instances to obtain first averages; and training the multi-task student instance to reproduce the first averages. ### Claim 6 (depends on 5) The method of claim 5, wherein the multi-task student instance is trained to reproduce at least two different probabilities for a given category on a given instance of training data. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of candidate multi-task teacher instances and the multi-task student instance each include an encoder configured to output context embedding vectors to the first task-specific layer and the second task-specific layer. ### Claim 8 A system comprising: a hardware processing unit; and a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the system to: access a multi-task student instance, the multi-task student instance being a trained model that has been trained using first outputs of one or more first multi-task teacher instances to train a first task-specific layer of the multi-task student instance and using second outputs of one or more second multi-task teacher instances to train a second task-specific layer of the multi-task student instance; receive input data; provide the input data to the multi-task student instance; obtain a first task-specific result produced by the first task-specific layer of the multi-task student instance; and output the first task-specific result, wherein the multi-task student instance, the one or more first multi-task teacher instances, and the one or more second multi-task teacher instances are instances of a multi-task machine learning model having one or more shared layers, the first task-specific layer, and the second task-specific layer. ### Claim 9 (depends on 8) The system of claim 8, wherein the multi-task student instance is a natural language processing model and the input data comprises input text. ### Claim 10 (depends on 9) The system of claim 9, wherein the one or more shared layers include a lexicon encoder and a transformer encoder that have been trained to map textual tokens into a vector space. ### Claim 11 (depends on 10) The system of claim 10, wherein the input text comprises a sentence and the first task-specific result characterizes a sentiment of the sentence as positive or negative. ### Claim 12 (depends on 10) The system of claim 10, wherein the input text comprises a pair of sentences and the first task-specific result characterizes a semantic similarity of the pair of sentences. ### Claim 13 (depends on 10) The system of claim 10, wherein the input text comprises a pair of sentences and the first task-specific result characterizes the pair of sentences as having an entailment relationship, a contradiction relationship, or a neutral relationship. ### Claim 14 (depends on 10) The system of claim 10, wherein the input text comprises a query and the first task-specific result comprises scores that reflect relevance of a plurality of documents to the query. ### Claim 15 (depends on 14) The system of claim 14, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: rank individual documents relative to one another based at least on the scores. ### Claim 16 One or more computer-readable storage media storing instructions which, when executed by one or more hardware processing units, cause the one or more hardware processing units to perform acts comprising: evaluating a plurality of candidate multi-task teacher instances of a multi-task machine learning model, each candidate multi-task teacher instance having one or more shared layers, a first task-specific layer that performs a first task, and a second task-specific layer that performs a second task; based at least on the evaluating, selecting one or more first multi-task teacher instances of the plurality of candidate multi-task teacher instances for the first task and one or more second multi-task teacher instances of the plurality of candidate multi-task teacher instances for the second task; and training a multi-task student instance of the multi-task machine learning model, the multi-task student instance being a trained model having the one or more shared layers, the first task-specific layer, and the second task-specific layer, the training comprising using first outputs of the one or more first multi-task teacher instances to train the first task-specific layer of the multi-task student instance and using second outputs of the one or more second multi-task teacher instances to train the second task-specific layer of the multi-task student instance. ### Claim 17 (depends on 16) The one or more computer-readable storage media of claim 16, wherein a pretraining stage has been performed on the one or more shared layers of the one or more first multi-task teacher instances and the one or more second multi-task teacher instances. ### Claim 18 (depends on 17) The one or more computer-readable storage media of claim 17, wherein the one or more shared layers comprise an encoder configured to map natural language tokens into token embeddings in a vector space. ### Claim 19 (depends on 18) The one or more computer-readable storage media of claim 18, wherein a tuning stage has been performed on the first task-specific layer and the second task-specific layer of the one or more first multi-task teacher instances and the first task-specific layer and the second task-specific layer of the one or more second multi-task teacher instances, the tuning stage using different task-specific sets of textual training data to tune the first task-specific layer and the second task-specific layer. ### Claim 20 (depends on 18) The one or more computer-readable storage media of claim 18, the encoder being a transformer encoder.