# US 2021/0342455 A1 · Image content obfuscation using a neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342455 A1
- Document ID: US20210342455A1
- Status: Published application
- Kind: A1
- Application number: 16/864,465
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Kamlesh Dattaram KSHIRSAGAR; Frank T. SEIDE
- Web page: https://patentgrep.com/patent/US20210342455A1

## CPC

- G06F 21/6209: {to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06F 21/602: {Providing cryptographic facilities or services}
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/20: in augmented reality scenes
- G06V 40/113: {Recognition of static hand signs}
- G06V 40/172: {Classification, e.g. identification}
- G06F 2221/2107: File encryption
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The technology described herein obfuscates image content using a local neural network and a remote neural network. The local network runs on a local computer system and a remote classifier runs in a remote computing system. Together, the local network and the remote classifier are able to classify images, while the image never leaves the local computer system. In aspects of the technology, the local network receives a local image and creates a transformed object. The transformed object may be generated by processing the image with a local neural network to generate a multidimensional array and then randomly shuffling data locations within a multidimensional array. The transformed object is communicated to the remote classifier in the remote computing system for classification. The remote classifier may not have the seed used to deterministically scramble the spatial arrangement of data within the multidimensional array.

## Claims

### Claim 1

A method of obfuscating image content comprising:
receiving an unlabeled image for classification;
processing the unlabeled image with a local neural network to generate a multidimensional array, the local neural network operating on a local computing system;
scrambling the multidimensional array deterministically to change a spatial arrangement of values within the multidimensional array to generate a transformed object;
communicating the transformed object to a remote classifier operating in a remote computing system;
receiving at the local computing system a classification for the unlabeled image produced by the remote classifier; and
saving at the local computing system the classification in association with the unlabeled image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the scrambling uses an undisclosed seed to govern a movement of the values within the multidimensional array.

### Claim 3 (depends on 2)

The method of claim 2, wherein the undisclosed seed is not communicated to the remote classifier.

### Claim 4 (depends on 1)

The method of claim 1, herein the remote classification model is trained on transformed objects generated by the local neural network processing labeled training images.

### Claim 5 (depends on 1)

The method of claim 1, wherein the local neural network comprises a convolutional pooling layer.

### Claim 6 (depends on 5)

The method of claim 5, wherein the transformed object comprises an output from each neuron in the convolutional pooling layer.

### Claim 7 (depends on 1)

The method of claim 1, wherein the local neural network does not assign a classification to an image.

### Claim 8

A method for obfuscating image content comprising:
receiving a transformed object generated from an unlabeled image by a local neural network, an arrangement of data within the transformed object having a altered spatial relationship to pixels within the unlabeled image;
processing the transformed object by a remote classifier residing in a remote computing system;
generating a classification for the unlabeled image using the remote classifier; and
communicating the classification for the unlabeled image to a local computing system that generated the transformed object.

### Claim 9 (depends on 8)

The method of claim 8, wherein the transformed object comprises an output from each neuron of a layer in the local neural network.

### Claim 10 (depends on 8)

The method of claim 8, wherein the altered spatial relationship to pixels within the unlabeled image is generated using a deterministic scramble function that produces a unique spatial rearrangement in response to receiving a seed value as input.

### Claim 11 (depends on 10)

The method of claim 10, wherein the remote the seed value is not available to the remote classifier.

### Claim 12 (depends on 10)

The method of claim 10, wherein the transformed object is not unscrambled before processing by the remote classifier.

### Claim 13 (depends on 9)

The method of claim 9, wherein the layer is a convolutional pooling layer.

### Claim 14 (depends on 8)

The method of claim 8, wherein the unlabeled image, the transformed object, and the classification are all associated with a common identification.

### Claim 15 (depends on 8)

The method of claim 8, wherein the remote classifier is trained using transformed objects received from the local neural network.

### Claim 16

A computer-storage media having computer-executable instructions embodied thereon that when executed by a computer processor causes a mobile computing device to perform a method of obfuscating image content comprising, the method comprising:
training a local neural network;
processing a training image with the local neural network to generate a multidimensional array, the local neural network operating on a local computing system;
shuffling the multidimensional array to change a spatial arrangement of values within the multidimensional array to generate a transformed object;
communicating the transformed object and a label associated with the training image to a remote computing system;
inputting the transformed object into a remote classifier to generate a classification of the training image;
using the label and the classification to retrain the remote classifier to form a trained remote classifier; and
storing the trained remote classifier in the remote computing system.

### Claim 17 (depends on 16)

The media of claim 16, wherein the transformed object comprises an output from each neuron of a layer of the local neural network.

### Claim 18 (depends on 16)

The media of claim 16, wherein the transformed object is not unscrambled before processing by the remote classifier.

### Claim 19 (depends on 16)

The media of claim 16, wherein the spatial arrangement of values is changed using a deterministic scramble function that produces a unique spatial rearrangement in response to receiving a seed value as input.

### Claim 20 (depends on 19)

The media of claim 19, wherein the remote the seed value is not available to the remote classifier.
