# US 2021/0334459 A1 · Robustness to adversarial behavior for text classification models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334459 A1
- Document ID: US20210334459A1
- Status: Published application
- Kind: A1
- Application number: 17/239,284
- Filed: 2021-04-23
- Published: 2021-10-28
- Earliest claimed: 2020-04-24
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Krishnamurthy Dvijotham; Anton Zhernov; Sven Adrian Gowal; Conrad Grobler; Robert Stanforth
- Web page: https://patentgrep.com/patent/US20210334459A1

## CPC

- G06F 40/216: using statistical methods
- G06F 40/279: Recognition of textual entities
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/247: Thesauruses; Synonyms
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 40/30: Semantic analysis
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a text classification machine learning model. One of the methods includes training a model having a plurality of parameters and configured to generate a classification of a text sample comprising a plurality of words by processing a model input that includes a combined feature representation of the plurality of words in the text sample, wherein the training comprises receiving a text sample and a target classification for the text sample; generating a plurality of perturbed combined feature representations; determining, based on the plurality of perturbed combined feature representations, a region in the embedding space; and determining an update to the parameters based on an adversarial objective that encourages the model to assign the target classification for the text sample for all of the combined feature representations in the region in the embedding space.

## Claims

### Claim 1

A method of training a machine learning model having a plurality of parameters and configured to generate a classification of a text sample comprising a plurality of words by processing a model input that includes a combined feature representation of the plurality of words in the text sample, wherein the combined feature representation is a representation in an embedding space,
wherein the training comprises:
receiving a training text sample and a target classification for the training text sample;
generating a plurality of perturbed combined feature representations, wherein each perturbed combined feature representation is a representation in the embedding space, and wherein generating each of the perturbed combined feature representations comprises:
selecting one or more words in the training text sample;
generating a perturbed training text sample by determining a respective perturbed word for each selected word; and
generating a combined feature representation of the perturbed training text sample;
determining, based on the plurality of perturbed combined feature representations, a region in the embedding space; and
determining an update to the parameters based on an adversarial objective that encourages the machine learning model to assign the target classification for the training text sample to all of the combined feature representations in the region in the embedding space.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining a perturbed word for a selected word comprises selecting a perturbed word that has a combined feature representation that is close to the combined feature representation of the selected word in the embedding space.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining a perturbed word for a selected word comprises selecting a perturbed word that is a synonym of the selected word in a thesaurus.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining a perturbed word for a selected word comprises processing the selected word using a sequence of one or more single-character edits, wherein the sequence of single-character edits comprises one or more of:
one or more insertions;
one or more deletions; or
one or more substitutions.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating a perturbed training text example comprises removing one or more of the selected words from the training text sample.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
each word in a vocabulary of words has a word embedding, and
generating a combined feature representation of the perturbed training text sample comprises combining word embeddings of one or more perturbed words that are in the vocabulary.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
each word in a vocabulary of words has a word embedding, and
generating the combined feature representation of the perturbed training text sample comprises combining word embeddings of unselected words in the respective training text sample that are in the vocabulary.

### Claim 8 (depends on 1)

The method of claim 1, wherein each combined feature representation of a text sample is a measure of central tendency of word embeddings of the words in the text sample.

### Claim 9 (depends on 1)

The method of claim 1, wherein the region in the embedding space is a convex hull of the perturbed combined feature representations.

### Claim 10 (depends on 1)

The method of claim 1, wherein determining an update to the parameters based on an adversarial objective that encourages the model to assign the target classification for all combined feature representations in the region in the embedding space comprises:
determining one or more worst combined feature representations in the region in the embedding space, wherein a worst combined feature representation has a large measure of error, wherein the measure of error for a combined feature representation is a measure of a difference between the target classification of the combined feature representation and a classification of the combined feature representation generated by the machine learning model; and
determining an update to the parameters using the worst combined feature representations.

### Claim 11 (depends on 10)

The method of claim 10, wherein determining a worst combined feature representation in the region in the embedding space comprises:
selecting a first combined feature representation in the region;
performing one or more gradient ascent steps to select respective intermediate combined feature representations in the region that have a larger measure of error than previous combined feature representations; and
selecting the worst combined feature representation after a final gradient ascent step.

### Claim 12 (depends on 10)

The method of claim 10, wherein determining an update to the parameters using the worst combined feature representations comprises:
processing the worst combined feature representations using the machine learning model; and
determining an update to the parameters using the measure of error of the worst combined feature representations.

### Claim 13 (depends on 1)

The method of claim 1, wherein:
each text sample comprises text that describes an application, and
the machine learning model is configured to classify a type of the application, wherein the model is either i) a multi-class classifier for a plurality of application types or ii) a binary classifier for a particular type of interest.

### Claim 14

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform a method of training a machine learning model having a plurality of parameters and configured to generate a classification of a text sample comprising a plurality of words by processing a model input that includes a combined feature representation of the plurality of words in the text sample, wherein the combined feature representation is a representation in an embedding space,
wherein the training comprises:
receiving a training text sample and a target classification for the training text sample;
generating a plurality of perturbed combined feature representations, wherein each perturbed combined feature representation is a representation in the embedding space, and wherein generating each of the perturbed combined feature representations comprises:
selecting one or more words in the training text sample;
generating a perturbed training text sample by determining a respective perturbed word for each selected word; and
generating a combined feature representation of the perturbed training text sample;
determining, based on the plurality of perturbed combined feature representations, a region in the embedding space; and
determining an update to the parameters based on an adversarial objective that encourages the machine learning model to assign the target classification for the training text sample to all of the combined feature representations in the region in the embedding space.

### Claim 15 (depends on 14)

The system of claim 14, wherein determining a perturbed word for a selected word comprises selecting a perturbed word that has a combined feature representation that is close to the combined feature representation of the selected word in the embedding space.

### Claim 16 (depends on 14)

The system of claim 14, wherein determining a perturbed word for a selected word comprises selecting a perturbed word that is a synonym of the selected word in a thesaurus.

### Claim 17 (depends on 14)

The system of claim 14, wherein determining a perturbed word for a selected word comprises processing the selected word using a sequence of one or more single-character edits, wherein the sequence of single-character edits comprises one or more of:
one or more insertions;
one or more deletions; or
one or more substitutions.

### Claim 18 (depends on 14)

The system of claim 14, wherein generating a perturbed training text example comprises removing one or more of the selected words from the training text sample.

### Claim 19

One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform a method of training a machine learning model having a plurality of parameters and configured to generate a classification of a text sample comprising a plurality of words by processing a model input that includes a combined feature representation of the plurality of words in the text sample, wherein the combined feature representation is a representation in an embedding space,
wherein the training comprises:
receiving a training text sample and a target classification for the training text sample;
generating a plurality of perturbed combined feature representations, wherein each perturbed combined feature representation is a representation in the embedding space, and wherein generating each of the perturbed combined feature representations comprises:
selecting one or more words in the training text sample;
generating a perturbed training text sample by determining a respective perturbed word for each selected word; and
generating a combined feature representation of the perturbed training text sample;
determining, based on the plurality of perturbed combined feature representations, a region in the embedding space; and
determining an update to the parameters based on an adversarial objective that encourages the machine learning model to assign the target classification for the training text sample to all of the combined feature representations in the region in the embedding space.

### Claim 20 (depends on 19)

The non-transitory computer storage media of claim 19, wherein determining a perturbed word for a selected word comprises selecting a perturbed word that has a combined feature representation that is close to the combined feature representation of the selected word in the embedding space.
