# US 12,591,752 B2 · Zero-shot domain transfer with a text-to-text model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,591,752 B2
- Document ID: US12591752B2
- Status: Granted
- Kind: B2
- Application number: 18/161,033
- Filed: 2023-01-27
- Published: 2024-08-01
- Granted: 2026-03-31
- Earliest claimed: 2023-01-27
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Stephanie Hyland; Aditya Nori; Fangyu Liu; Fernando Perez Garcia; Qianchu Liu; Hoifung Poon; Javier Alvarez-Valle; Naoto Usuyama; Ozan Oktay; Sheng Zhang; Shruthi Jaisimha Bannur; Tristan Josef Naumann
- Web page: https://patentgrep.com/patent/US12591752B2

## CPC

- G06F 40/56: Natural language generation
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- H04L 51/18: Commands or executable codes
- H04L 51/234: for tracking messages
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Example solutions for zero-shot domain transfer with a text-to-text model train a text-to-text model for a target domain using unlabeled in-domain text training data, and concurrently train the model using labeled general-domain task training data. The in-domain training comprises masked language modeling (MLM) training, and the task training comprises both natural language generation (NLG) training and natural language understanding (NLU) training. The NLG training comprises natural language inference (NLI) training and the NLU training comprises summarization training. The trained model acquires domain-specific task competency, sufficient to perform a language task within the target domain. Suitable target domains include radiology, biomedical, and other medical, legal, and scientific domains. This approach leverages large volumes of general-domain task training data and plentiful unlabeled in-domain text, even as labeled in-domain training data may be unavailable or prohibitively expensive for certain specialized domains.

## Claims

### Claim 1

A system comprising:
a processor; and
a computer storage medium storing instructions that are operative upon execution by the processor to:
train a text-to-text model for a specialized target domain, training material for the specialized target domain being limited to a particular subject matter, wherein:
the text-to-text model has an encoder-decoder generative language framework; and
the training comprises simultaneously performing:
masked language modeling (MLM) training using unlabeled text from the specialized target domain, wherein the text-to-text model acquires domain-specific knowledge;
task training using labeled task training data, resulting in the text-to-text model acquiring transferable task knowledge, wherein the task training comprises:
natural language generation (NLG) training, including natural language inference (NLI) generation tasks; and
natural language understanding (NLU) training, including summarization classification tasks; and
the MLM training and the task training collectively cause the text-to-text model to acquire domain-specific task competency; and
use the trained text-to-text model to perform a language task within the specialized target domain.

### Claim 2 (depends on 1)

The system of claim 1, wherein the labeled task training data comprises general domain data.

### Claim 3 (depends on 1)

The system of claim 1, wherein the NLI generation tasks comprise summarization.

### Claim 4 (depends on 1)

The system of claim 1, wherein the instructions are further operative to:
generate, with the text-to-text model, additional NLI training data in the specialized target domain; and
further train the text-to-text model with the additional NLI training data.

### Claim 5 (depends on 1)

The system of claim 1, wherein the specialized target domain comprises a knowledge domain selected from a list consisting of: medical, radiology, biomedical, law, finance, mathematics, chemistry physics, and engineering.

### Claim 6 (depends on 1)

The system of claim 1, wherein the task training comprises a task selected from a list consisting of: summarization, data augmentation, generating target domain training data for a machine learning (ML) model different than the text-to-text model, document retrieval, classification, determining text similarity, and synthetic data generation.

### Claim 7 (depends on 1)

The system of claim 1, wherein the training task comprises a task to generate text or answer a question.

### Claim 8

A computerized method comprising:
training a text-to-text model for a specialized target domain, training material for the specialized target domain being limited to a particular subject matter, wherein:
the text-to-text model has an encoder-decoder generative language framework; and
the training comprises simultaneously performing:
masked language modeling (MLM) training using unlabeled text from the specialized target domain, wherein the text-to-text model acquires domain-specific knowledge;
task training using labeled task training data, resulting in the text-to-text model acquiring transferrable task knowledge, wherein the task training comprises:
natural language generation (NLG) training, including natural language inference (NLI) generation tasks; and
natural language understanding (NLU) training, including summarization classification tasks; and
the MLM training and the task training collectively cause the text-to-text model to acquire domain-specific task competency; and
using the trained text-to-text model to perform a language task within the specialized target domain.

### Claim 9 (depends on 8)

The method of claim 8, wherein the labeled task training data comprises general domain data.

### Claim 10 (depends on 8)

The method of claim 8, wherein the NLI generation tasks comprise summarization.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
generating, with the text-to-text model, additional NLI training data in the specialized target domain; and
further training the text-to-text model with the additional NLI training data.

### Claim 12 (depends on 8)

The method of claim 8, wherein the specialized target domain comprises a knowledge domain selected from a list consisting of: medical, radiology, biomedical, law, finance, mathematics, chemistry physics, and engineering.

### Claim 13 (depends on 8)

The method of claim 8, wherein the task training comprises a task selected from a list consisting of: summarization, data augmentation, generating target domain training data for a machine learning (ML) model different than the text-to-text model, document retrieval, classification, determining text similarity, and synthetic data generation.

### Claim 14 (depends on 8)

The method of claim 8, wherein the text-to-text model comprises a text-to-text transfer transformer (T5) model.

### Claim 15

A computer storage device having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising:
training a text-to-text model for a specialized target domain, training material for the specialized target domain being limited to a particular subject matter, wherein:
the text-to-text model has an encoder-decoder generative language framework; and
the training comprising simultaneously performing:
masked language modeling (MLM) training using unlabeled text from the specialized target domain, wherein the text-to-text model acquires domain-specific knowledge;
task training using labeled task training data that is not limited to a particular subject matter, wherein the task training comprises
natural language generation (NLG) training, including natural language inference (NLI) generation tasks; and
natural language understanding (NLU) training, including summarization classification tasks; and
the MLM training and the task training collectively cause the text-to-text model to acquire domain-specific task competency; and
using the trained text-to-text model to perform a language task within the target domain.

### Claim 16 (depends on 15)

The computer storage device of claim 15, wherein the NLI generation tasks comprise summarization.

### Claim 17 (depends on 15)

The computer storage device of claim 15, wherein the operations further comprise:
generating, with the text-to-text model, additional NLI training data in the specialized target domain; and
further training the text-to-text model with the additional NLI training data.

### Claim 18 (depends on 15)

The computer storage device of claim 15, wherein the specialized target domain comprises a knowledge domain selected from a list consisting of: medical, radiology, biomedical, law, finance, mathematics, chemistry physics, and engineering.

### Claim 19 (depends on 15)

The computer storage device of claim 15, wherein the task training comprises a task selected from a list consisting of: summarization, data augmentation, generating target domain training data for a machine learning (ML) model different than the text-to-text model, document retrieval, classification, determining text similarity, and synthetic data generation.

### Claim 20 (depends on 15)

The computer storage device of claim 15, wherein the labeled task training data comprises an indication of a training task to perform, a training input text, and a training target output.