# US 12,182,308 B2 · Removal of sensitive data from documents for use as training sets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,308 B2
- Document ID: US12182308B2
- Status: Granted
- Kind: B2
- Application number: 17/309,198
- Filed: 2019-11-07
- Published: 2021-12-23
- Granted: 2024-12-31
- Earliest claimed: 2018-11-07
- Assignees: ServiceNow Canada Inc.
- Applicants: Servicenow Canada Inc.
- Inventors: Archy Otto De Berker; Philippe Guay; Dominique Tourillon; Etienne Marcotte
- Web page: https://patentgrep.com/patent/US12182308B2

## CPC

- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06V 10/82: using neural networks
- G06V 30/19167: {Active pattern learning}
- G06V 30/19173: {Classification techniques}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06F 2221/2143: Clearing memory, e.g. to prevent the data from being stolen
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30176: Document
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods relating to the replacement or removal of sensitive data in images of documents. An initial image of a document with sensitive data is received at an execution module and changes are made based on the execution module's training. The changes include replacing or effectively removing the sensitive data from the image of the document. The resulting sanitized image is then sent to a user for validation of the changes. The feedback from the user is then used in training the execution module to refine its behaviour when applying changes to other initial images of documents. To train the execution module, training data sets of document images with sensitive data manually tagged by users are used. The execution module thus learns to identify sensitive data and its submodules replace that sensitive data with suitable replacement data. The feedback from the user works to improve the resulting sanitized images from the execution module.

## Claims

### Claim 1

A method for replacing sensitive data in an image of a document, the method being executed by at least one processor, the method comprising:
a) receiving an initial image of said document;
b) applying at least one change to said initial image to result in a sanitized image of said document by:
using a trained convolutional neural network, determining areas comprising sensitive data in the initial image of the document by delineating the areas;
using the trained convolutional neural network, determining at least one type of sensitive data in the areas comprising the sensitive data; and
using a trained generative adversarial network, generating, based on the at least one type of sensitive data, suitable replacement data for the sensitive data for applying the at least one change to obtain the sanitized image of said document;
c) sending said sanitized image to a user for validation;
d) receiving user feedback regarding said sanitized image;
e) using said user feedback to further train at least one of said trained convolutional neural network and said trained generative adversarial network,
wherein said feedback used in step e) comprises at least one addition of a delineation of sensitive data that has not been removed or replaced in said initial image;
wherein said at least one change includes an effective removal or a replacement of said sensitive data in said image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein said user feedback comprises corrections to said change.

### Claim 3 (depends on 1)

The method according to claim 1, wherein said change comprises at least one of:
adding at least one bounding box to a cluster of sensitive data in said initial image, sensitive data within said at least one bounding box being for effective removal or replacement with non-sensitive replacement data;
masking at least one cluster of sensitive data in said document such that said sensitive data is obfuscated or is otherwise unreadable; and
placing a border around at least one element in said initial image, said at least one element comprising said sensitive data such that said at least one element is for effective removal or replacement in said image.

### Claim 4 (depends on 1)

The method according to claim 1, wherein said feedback consists of an approval or a rejection of said at least one change.

### Claim 5 (depends on 1)

The method according to claim 1, wherein said sensitive data of at least one specific type of data comprises at least one of:
names of persons;
addresses of persons;
gender;
confidential information;
physical characteristics;
financial information;
health related information;
business related information;
names of corporate entities;
addresses of corporate entities;
dates;
credit card numbers;
bank account data;
social security/social insurance numbers;
personal identification data; and
blood types.

### Claim 6 (depends on 1)

The method according to claim 1, wherein said sensitive data is replaced by replacement data and elements of said replacement data are randomly selected.

### Claim 7 (depends on 1)

The method according to claim 1, wherein said sensitive data is replaced by replacement data and at least portion of said replacement data is randomly generated.

### Claim 8 (depends on 1)

The method according to claim 1, wherein said sensitive data is replaced by replacement data and said replacement data is a block that effectively removes said sensitive data from said image.

### Claim 9

A system for use in replacing sensitive data in a document, the system comprising:
a non-transitory storage medium storing computer-readable instructions thereon;
and at least one processor operatively connected to the non-transitory storage medium, the at least one processor, upon executing the computer-readable instructions, being configured for:
receiving an initial image of said document and for applying at least one change to said initial image to result in a sanitized image by:
using a trained convolutional neural network, determining areas comprising sensitive data in the initial image of the document by delineating the areas;
using the trained convolutional neural network, determining at least one type of sensitive data in the areas comprising the sensitive data; and
using a trained generative adversarial network, generating, based on the at least one type of sensitive data, suitable replacement data for the sensitive data for applying the at least one change to obtain the sanitized image of said document;
sending said sanitized image of said document to a user for validation and for receiving feedback from said user;
wherein:
said feedback comprises at least one addition of a delineation of sensitive data that has not been removed or replaced in said initial image;
said feedback is used for further training at least one of said trained convolutional neural network and said trained generative adversarial network; and
said at least one change includes an effective removal or a replacement of said sensitive data in said document.

### Claim 10 (depends on 9)

The system according to claim 9, further comprising storing said feedback received from said user in said non-transitory storage medium.

### Claim 11 (depends on 9)

The system according to claim 9, further comprising receiving said feedback from said user and continuously training said at least one of said trained convolutional neural network and said trained generative adversarial network for adjusting a behaviour of said at least one said trained convolutional neural network and said trained generative adversarial network based on said feedback.

### Claim 12 (depends on 9)

The system according to claim 9, wherein said sensitive data of at least one specific type of data comprises at least one of:
names of persons;
addresses of persons;
gender;
confidential information;
physical characteristics;
financial information;
health related information;
business related information;
names of corporate entities;
addresses of corporate entities;
dates;
credit card numbers;
bank account data;
social security/social insurance numbers;
personal identification data; and
blood types.

### Claim 13 (depends on 9)

The system according to claim 9, wherein said document is received and processed by said system as a digital image of said document.

### Claim 14

A method for replacing sensitive data in a document, the method being executed by at least one processor, the method comprising:
a) receiving an initial version of said document;
b) applying at least one change to said initial version to result in a sanitized version of said document by:
using a trained convolutional neural network, determining areas comprising sensitive data in the initial version of the document by delineating the areas;
using the trained convolutional neural network, determining at least one type of sensitive data in the areas comprising the sensitive data; and
using a trained generalive adversarial network, generating, based on the at least one type of sensitive data, suitable replacement data for the sensitive data for applying the at least one change to obtain the sanitized version of said document;
c) sending said sanitized version to a user for validation;
d) receiving user feedback regarding said sanitized version;
e) using said user feedback to further train at least one of said trained convolutional neural network and said trained generative adversarial network;
wherein said feedback used in step e) comprises at least one addition of a delineation of sensitive data that has not been removed or replaced in said initial version;
wherein said at least one change includes an effective removal or a replacement of said sensitive data in said document.