US 12,591,752 B2 · Zero-shot domain transfer with a text-to-text model

Abstract

Example solutions for zero-shot domain transfer with a text-to-text model train a text-to-text model for a target domain using unlabeled in-domain text training data, and concurrently train the model using labeled general-domain task training data. The in-domain training comprises masked language modeling (MLM) training, and the task training comprises both natural language generation (NLG) training and natural language understanding (NLU) training. The NLG training comprises natural language inference (NLI) training and the NLU training comprises summarization training. The trained model acquires domain-specific task competency, sufficient to perform a language task within the target domain. Suitable target domains include radiology, biomedical, and other medical, legal, and scientific domains. This approach leverages large volumes of general-domain task training data and plentiful unlabeled in-domain text, even as labeled in-domain training data may be unavailable or prohibitively expensive for certain specialized domains.

Claims20

Dependent claims

A system comprising:

a processor;
and
a computer storage medium storing instructions that are operative upon execution by the processor to:
train a text-to-text model for a specialized target domain, training material for the specialized target domain being limited to a particular subject matter, wherein:
the text-to-text model has an encoder-decoder generative language framework;
and
the training comprises simultaneously performing:
masked language modeling (MLM) training using unlabeled text from the specialized target domain, wherein the text-to-text model acquires domain-specific knowledge;
task training using labeled task training data, resulting in the text-to-text model acquiring transferable task knowledge, wherein the task training comprises:
natural language generation (NLG) training, including natural language inference (NLI) generation tasks;
and
natural language understanding (NLU) training, including summarization classification tasks;
and
the MLM training and the task training collectively cause the text-to-text model to acquire domain-specific task competency;
and
use the trained text-to-text model to perform a language task within the specialized target domain.

13 more elements

Dependencies are inferred from the claim text.

Zero-shot domain transfer with a text-to-text model

Life of this patent

Zero-shot domain transfer with a text-to-text model

Life of this patent

Abstract

Claims20