# US 2021/0342546 A1 · Systems and methods for a privacy preserving text representation learning framework

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342546 A1
- Document ID: US20210342546A1
- Status: Published application
- Kind: A1
- Application number: 17/245,774
- Filed: 2021-04-30
- Published: 2021-11-04
- Earliest claimed: 2020-04-30
- Assignees: Arizona Board of Regents on behalf of Arizona State University
- Inventors: Ghazaleh Beigi; Kai Shu; Ruocheng Guo; Suhang Wang; Huan Liu
- Web page: https://patentgrep.com/patent/US20210342546A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 18/21322: {Rendering the within-class scatter matrix non-singular}
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06F 18/21326: {involving optimisations, e.g. using regularisation techniques}
- G06N 3/047: Probabilistic or stochastic networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Various embodiments of a computer-implemented system which learns textual representations while filtering out potentially personally identifying data and retaining semantic meaning within the textual representations are disclosed herein.

## Claims

### Claim 1

A method of generating a modified latent text representation for a document, comprising:
utilizing a processor in communication with a tangible storage medium storing instructions that are executed by the processor to perform operations comprising:
generating an initial latent representation representative of text in a document;
inferring an amount of noise to be added to the initial latent representation by finding a privacy budget value that minimizes a loss between a predicted semantic label and a ground truth semantic label for the initial latent representation and maximizes a loss between a predicted private attribute label and a ground truth private attribute label for the initial latent representation; and
adding the amount of noise to the initial latent text representation to generate a modified latent text representation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the initial latent representation is generated using an auto-encoder trained to generate the initial latent representation from the text in the document.

### Claim 3 (depends on 2)

The method of claim 2, further comprising training the autoencoder by:
generating an encoded latent representation representative of the text in the document by applying an encoder to the document;
constructing a reconstructed document including reconstructed text representative of the text in the encoded latent representation by applying a decoder to the encoded latent representation; and
identifying a plurality of autoencoder parameters that minimize a loss between the text in the document and the reconstructed text in the reconstructed document.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
optimizing a set of semantic discriminator classifier parameters and a privacy budget value that minimize the loss between the predicted semantic label and the ground truth semantic label for the initial latent representation.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
adding a first amount of noise to the initial latent representation to generate a modified latent representation.

### Claim 6 (depends on 4)

The method of claim 4, further comprising:
generating the predicted semantic label by applying a classifier to the modified latent representation;
minimizing a loss between the predicted semantic label and the ground truth semantic label; and
identifying the set of semantic discriminator classifier parameters associated with the lowest loss value between the predicted semantic label and the ground truth semantic label.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
selecting the privacy budget value that is associated with a lowest loss value between the predicted semantic label and the ground truth semantic label and that is associated with a highest loss value between the predicted private attribute label and a ground truth private attribute label.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
optimize a set of private attribute discriminator parameters and a privacy budget value that maximize the loss between the predicted private attribute label and the ground truth private attribute label for the initial latent representation.

### Claim 9 (depends on 8)

The method of claim 8, wherein the optimization of the set of private attribute discriminator parameters is modeled as a minmax game.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
generating the predicted private attribute label by applying a classifier to the modified latent representation;
maximizing a loss between the predicted private attribute label and the ground truth private attribute label; and
selecting the set of private attribute discriminator classifier parameters associated with the lowest loss value between the predicted private attribute label and the ground truth private attribute label.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
determining the amount of noise to add based on the privacy budget value by sampling a value r from a uniform distribution such that:
s
⁡
(
i
)
=
-
Δ
ϵ
×
sgn
⁡
(
r
)
⁢
ln
⁡
(
1
-
2
|
r
|
)
,
i
=
1
,
2
,
.
.
.
⁢
,
d
wherein ϵ is a privacy budget, Δ is an L1-sensitivity of the initial latent representation, d is a dimension of the initial latent representation, s is a noise vector and s(i) is the i-th element for noise vector s.

### Claim 12 (depends on 1)

The method of claim 1, wherein the step of finding the privacy budget value is iteratively repeated until convergence.

### Claim 13 (depends on 12)

The method of claim 12, wherein the process of adding the amount of noise to the initial latent text representation to generate a modified latent text representation runs concurrently with finding the privacy budget value.

### Claim 14

A computer system for generating a modified latent text representation for a document, comprising:
at least one processor in communication with a memory and operable for execution of a plurality of modules, the plurality of modules including:
an auto-encoder configured to generate an initial latent representation representative of text in a document;
a noise adder module configured to receive the initial latent representation and add an amount of noise to the initial latent representation to generate a modified latent text representation based on a privacy budget value;
a semantic meaning discriminator module configured to optimize a set of semantic discriminator classifier parameters and the privacy budget value such that a loss is minimized between the predicted semantic label and the ground truth semantic label for the initial latent representation; and
a private attribute discriminator module configured to optimize a set of private attribute discriminator parameters and the privacy budget value such that a loss is maximized between the predicted private attribute label and the ground truth private attribute label for the initial latent representation.

### Claim 15 (depends on 14)

The computer system of claim 14, wherein the auto-encoder module is configured to:
generate an encoded latent representation representative of the text in the document by applying an encoder to the document;
construct a reconstructed document including reconstructed text representative of the text in the encoded latent representation by applying a decoder to the encoded latent representation; and
identify a plurality of autoencoder parameters that minimize a loss between the text in the document and the reconstructed text in the reconstructed document.

### Claim 16 (depends on 14)

The computer system of claim 14, wherein the semantic meaning discriminator module is configured to:
generate the predicted semantic label by applying a first classifier to the modified latent representation;
minimize the loss between the predicted semantic label and the ground truth semantic label; and
identify the set of semantic discriminator classifier parameters associated with the lowest loss value between the predicted semantic label and the ground truth semantic label.

### Claim 17 (depends on 16)

The computer system of claim 16, wherein the first classifier is implemented using a recurrent neural network that takes the set of semantic discriminator classifier parameters and the modified latent text representation as input.

### Claim 18 (depends on 14)

The computer system of claim 14, wherein the private attribute discriminator module is configured to:
generate the predicted private attribute label by applying a second classifier to the modified latent representation;
maximize a loss between the predicted private attribute label and the ground truth private attribute label; and
select the set of private attribute discriminator classifier parameters associated with the lowest loss value between the predicted private attribute label and the ground truth private attribute label.

### Claim 19 (depends on 18)

The computer system of claim 18, wherein the second classifier is implemented using a recurrent neural network that takes the set of private attribute discriminator classifier parameters and the modified latent text representation as input.

### Claim 20 (depends on 18)

The computer system of claim 18, wherein the privacy budget value is associated with a lowest loss value between the predicted semantic label and the ground truth semantic label and a highest loss value between the predicted private attribute label and a ground truth private attribute label.
