US 11,538,467 B1Granted

Calibrated noise for text modification

AssigneeAmazon Technologies, Inc.

InventorOluwaseyi Oluwafemi Feyisetan

Life of this patent

Estimating the term…

Loading patent…

Abstract

Devices and techniques are generally described for calibrating noise for natural language data modification. In various examples, first data representing a natural language input may be identified. A first vector representation of a first word of the first data may be determined. Sensitivity data may be determined for the first vector representation based at least in part on a first density of one or more vector representations adjacent to the first vector representation in an embedding space. In some examples, a first noise vector may be determined based at least in part on the sensitivity data. A first modified vector representation may be generated by adding the first noise vector to the first vector representation. A second word may be determined based at least in part on the first modified vector representation. Modified first data may be generated by replacing the first word with the second word.

Claims20

Dependent claims

A method comprising:

identifying first text data representing a natural language input, wherein the first text data is included in a training data set for a first natural language machine learning model;
determining a first word of the first text data for input into a randomization function R;
determining a first vector representation of the first word in an embedding space, the embedding space comprising representations of a plurality of words;
determining a first average distance between the first vector representation of the first word and a first plurality of vector representations of k closest words to the first vector representation in the embedding space;
determining a first local sensitivity of the randomization function R that limits a magnitude of modification of the first vector representation, wherein the first local sensitivity is determined using the first average distance;
determining a first noise vector N by sampling from a Laplace distribution, wherein the first noise vector Nis limited by the first local sensitivity;
generating a second vector representation by adding the first noise vector N to the first vector representation of the first word;
determining a third vector representation among the first plurality of vector representations that is closest to the second vector representation in the embedding space;
determining a second word associated with the third vector representation;
generating, by a first computing device, second text data by replacing the first word of the first text data with the second word
sending the second text data to a remote computing device;
receiving, by the first computing device from the remote computing device, first data comprising the second text data and a class label;
and
updating at least one parameter of the first natural language machine learning model using the first data.

11 more elements

A method comprising:

determining, by at least one first computing device, a first vector representation of a first word of first data, the first data representing a natural language input for natural language processing;
determining second data for the first vector representation based at least in part on a first average distance from the first vector representation to a first plurality of vector representations of k words in an embedding space, wherein the second data controls an amount of noise used to modify the first vector representation;
determining a first noise vector based at least in part on the second data;
generating a first modified vector representation using the first noise vector and the first vector representation;
determining a second word based at least in part on the first modified vector representation;
generating modified first data by replacing the first word with the second word in the first data;
and
updating, using the modified first data, at least one parameter value of a first natural language processing machine learning model.

5 more elements

A system comprising:

at least one processor;
and
at least one non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, program the at least one processor to:
determine a first vector representation of a first word of first data, the first data representing a natural language input for natural language processing;
determine second data for the first vector representation based at least in part on a first average distance from the first vector representation to a first plurality of vector representations of k words in an embedding space, wherein the second data controls an amount of noise used to modify the first vector representation;
determine a first noise vector based at least in part on the second data;
generate a first modified vector representation using the first noise vector and the first vector representation;
determine a second word based at least in part on the first modified vector representation;
generate modified first data by replacing the first word with the second word in the first data;
and
update, using the modified first data, at least one parameter value of a first natural language processing machine learning model.

8 more elements

Dependencies are inferred from the claim text.