Methods and system for context-preserving sensitive data anonymization
AssigneeProofpoint, Inc.
InventorsKarl Felix Joehnk, Romain Loic Choukroun
Life of this patent
Estimating the term…
Abstract
Systems and methods for privacy-preserving transformer model training are provided. The system includes one or more data repositories in a computer network or cloud infrastructure having data stored therein. The system anonymizes the data in the one or more documents, and trains a transformer model on the data outside of the network. The data includes sensitive information. Anonymizing the data is includes extracting the data from the one or more documents and irreversibly transforming the data in the one or more documents into context-preserving tensors. Training the transformer model on the data comprises using the context-preserving tensors instead of the data to train the transformer model on the data.
Claims22
A method for privacy-preserving transformer model training comprising:
- extracting data from one or more documents in one or more data repositories in a computer network or cloud infrastructure, the data comprising sensitive information;
- anonymizing the data by irreversibly transforming the data into context-preserving tensors, wherein irreversibly transforming the data into the context-preserving tensors comprises utilizing a transformer model for deep learning of the data by:
- determining, by a first set of layers of the transformer model and based on a non-square matrix representation of the data, gradients, wherein the gradients are frozen for processing by a second set of layers of the transformer model, and
- fine-tuning, by the second set of layers of the transformer model, the gradients;
- and
- training the transformer model on the data using the context-preserving tensors.
A privacy-preserving system for transformer model training comprising:
- one or more data repositories in a computer network or cloud infrastructure having data stored therein, the data comprising sensitive information;
- one or more non-transitory computer-readable media storing instructions that, when executed by the privacy-preserving system cause the privacy-preserving system to:
- anonymize the data in one or more documents in the one or more data repositories in the computer network or the cloud infrastructure by:
- extracting the data from the one or more documents and
- irreversibly transforming the data in the one or more documents into context-preserving tensors, wherein irreversibly transforming the data into the context-preserving tensors comprises utilizing a transformer model for deep learning of the data by:
- determining, by a first set of layers of the transformer model and based on a non-square matrix representation of the data, gradients, wherein the gradients are frozen for processing by a second set of layers of the transformer model, and
- fine-tuning, by the second set of layers of the transformer model, the gradients;
- and
- train the transformer model on the data using the context-preserving tensors instead of the data to train the transformer model on the data.
Dependencies are inferred from the claim text.