# US 11,526,626 B2 · Facial anonymization with consistent facial attribute preservation in video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,526,626 B2
- Document ID: US11526626B2
- Status: Granted
- Kind: B2
- Application number: 16/926,310
- Filed: 2020-07-10
- Published: 2022-01-13
- Granted: 2022-12-13
- Earliest claimed: 2020-07-10
- Assignees: De-Identification Ltd.
- Applicants: De-Identification Ltd.
- Inventors: Eliran Kuta; Sella Blondheim; Gil Perry; Yoav Hacohen; Amitay Nachmani; Naama Antebi; Matan Ben-Yosef; Or Gorodissky
- Web page: https://patentgrep.com/patent/US11526626B2

## CPC

- G06T 9/002: {using neural networks}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 40/172: {Classification, e.g. identification}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

To consistently anonymize video, an identity-preserving encoder may be trained with a decoder to preserve identity information unique to an individual in an encoded space. The identity-preserving encoder may encode source video images of a source identity to generate identity-preserving encoded data representing the source identity. A de-identification engine may generate de-identifying encoded data representing a different target identity. A transformation may be generated based on a difference between the identity-preserving and de-identifying encoded data, representing swapping the source identity with the target identity. The identity may be consistently swapped from the source to target in the encoded space by applying the same transformation to the identity-preserving encoded data for the source images. The swapped identity encoded data may be decoded to generate consistently anonymized output images representing the non-identity information of the source images with the target identity, for all the output video images.

## Claims

### Claim 1

A method for consistently anonymizing a source video comprising:
using an identity-preserving encoder, trained with a decoder to preserve identity information unique to an individual in an encoded space, encoding one or more source images in a source video of a person with a source identity to generate identity-preserving encoded data representing the source identity information in the encoded space;
using a de-identification engine, generating de-identifying encoded data representing information for a target identity different than the source identity in the encoded space,
wherein the de-identification engine includes a de-identification encoder that is trained to preserve non-identity information not unique to an individual, but is not trained to, or is trained not to, preserve the identity information unique to an individual,
wherein the de-identification encoder is a neural network trained to optimize weights of the neural network as:
w
′
=
argmin
w
⁢
∑
x
d
⁡
(
A
⁡
(
D
⁡
(
E
w
(
x
)
)
)
-
A
⁡
(
x
)
)
where x is an input image, Ew is an encoder with a set of candidate weights w, A is a non-identifying attribute function, and d is a difference function;
generating a transformation based on a difference between the identity-preserving encoded data and the de-identifying encoded data, the difference representing the removal of the source identity information and the addition of the target identity information;
swapping the identity from the source to the target in the encoded space by applying the same transformation to the identity-preserving encoded data for a plurality of source images causing the removal of the source identity information and the addition of the same target identity information for all of the plurality of source images; and
decoding the swapped identity encoded data to generate a plurality of output images representing the non-identity information from the plurality of source images and the same target identity, but not the source identity, information, for all the plurality of output images in an output video.

### Claim 2 (depends on 1)

The method of claim 1 comprising training the de-identification encoder with the decoder by a neural network that results in a sufficient loss of identifying information.

### Claim 3 (depends on 1)

The method of claim 1 comprising using the de-identification engine to search a database of pre-computed anonymous target images for a target image of an anonymous person with an anonymous identity that is different than, but that, based on smallest difference algorithm or a machine learning to match identities, most closely matches the source identity, and encode the target image with an identity-preserving encoder to generate the de-identifying encoded data.

### Claim 4 (depends on 3)

The method of claim 3, wherein when geometry of identifying information in the target image and source image substantially match, comprising using the identity-preserving encoded data for a single source image and the de-identifying encoded data for a single target image in the difference.

### Claim 5 (depends on 3)

The method of claim 3, wherein when geometry of identifying information in the target image and source images do not match, comprising warping the target image or the source image to have the same or approximately similar geometry as each other.

### Claim 6 (depends on 3)

The method of claim 3, wherein when geometry of identifying information in the target image and source images do not match, comprising averaging the identity-preserving encoded data for multiple source images and averaging the de-identifying encoded data for multiple target images in the difference.

### Claim 7 (depends on 6)

The method of claim 6 comprising generating the averaged de-identifying encoded data by initially decoding the averaged identity-preserving encoded data and re-encoding the decoded data with the de-identification encoder.

### Claim 8 (depends on 1)

The method of claim 1 comprising:
encoding, using the identity-preserving encoder, multiple source images of the source video and averaging the encoded representation of the multiple source frames to generate the identity-preserving encoded data; and
using the de-identification engine to obtain or search a database of pre-computed anonymous target images for a target video of an anonymous person with an anonymous identity that is different than, but that, based on smallest difference algorithm or a machine learning to match identities, most closely matches the source identity, encode multiple target images of the target video with an encoder and average the encoded representation of the multiple target frames to generate the de-identifying encoded data.

### Claim 9 (depends on 1)

The method of claim 1 comprising training the de-identification engine to perturb identity information unique to an individual in the encoded space.

### Claim 10 (depends on 1)

The method of claim 1 comprising iteratively repeating swapping the identity from a first target to a second target in the encoded space by applying a new difference for each identity swap to cause the removal of the first target identity information and the addition of the second target identity information for all of the plurality of source images.

### Claim 11

A method for consistently anonymizing a source video comprising:
using an identity-preserving encoder, trained with a decoder to preserve identity information unique to an individual in an encoded space, encoding one or more source images in a source video of a person with a source identity to generate identity-preserving encoded data representing the source identity information in the encoded space;
using a de-identification engine, generating de-identifying encoded data representing information for a target identity different than the source identity in the encoded space,
wherein the de-identification engine includes a de-identification encoder that is trained to preserve non-identity information not unique to an individual, but is not trained to, or is trained not to, preserve the identity information unique to an individual,
wherein the identity-preserving encoder is a neural network trained to optimize weights of the neural network as:
w
′
=
argmin
w
⁢
∑
x
d
⁡
(
D
⁡
(
E
w
(
x
)
)
-
x
)
where x is an input image, Ew is an encoder with a set of candidate weights w, and d is a difference function.

### Claim 12

A system for consistently anonymizing a source video, comprising:
one or more memories to store a source video comprising a plurality of source images of a person with a source identity; and
one or more processors configured to:
execute an identity-preserving encoder and decoder trained to preserve identity information unique to an individual in an encoded space, the identity-preserving encoder executed to encode one or more of the source images in the source video of the person with a source identity to generate identity-preserving encoded data representing the source identity information in the encoded space,
wherein the de-identification engine includes a de-identification encoder that is trained to preserve non-identity information not unique to an individual, but is not trained to, or is trained not to, preserve the identity information unique to an individual,
wherein the de-identification encoder is a neural network trained to optimize weights of the neural network as:
w
′
=
argmin
w
⁢
∑
x
d
⁡
(
D
⁡
(
E
w
(
x
)
)
-
x
)
where x is an input image, Ew is an encoder with a set of candidate weights w, A is a non-identifying attribute function, and d is a difference function,
execute a de-identification engine to generate de-identifying encoded data representing information for a target identity different than the source identity in the encoded space,
compute a transformation based on a difference between the identity-preserving encoded data and the de-identifying encoded data, the difference representing the removal of the source identity information and the addition of the target identity information,
swap the identity from the source to the target in the encoded space by applying the same transformation to the identity-preserving encoded data for a plurality of source images causing the removal of the source identity information and the addition of the same target identity information for all of the plurality of source images, and
execute the decoder to decode the swapped identity encoded data to generate a plurality of output images representing the non-identity information from the plurality of source images and the same target identity, but not the source identity, information, for all the plurality of output images in an output video.

### Claim 13 (depends on 12)

The system of claim 12, wherein the de-identification encoder is trained with the decoder by a neural network that results in a sufficient loss of identifying information.

### Claim 14 (depends on 12)

The system of claim 12, wherein the de-identification engine is executed to search a database of pre-computed anonymous target images for a target image of an anonymous person with an anonymous identity that is different than, but that, based on smallest difference algorithm or a machine learning to match identities, most closely matches the source identity, and encode the target image with an identity-preserving encoder to generate the de-identifying encoded data.

### Claim 15 (depends on 12)

The system of claim 12, wherein the one or more processors are configured to:
execute the identity-preserving encoder to encode multiple source images of the source video and average the encoded representation of the multiple source frames to generate the identity-preserving encoded data, and
execute the de-identification engine to obtain or search a database of pre-computed anonymous target images for a target video of an anonymous person with an anonymous identity that is different than, but that, based on smallest difference algorithm or a machine learning to match identities, most closely matches the source identity, encode multiple target images of the target video with an encoder and average the encoded representation of the multiple target frames to generate the de-identifying encoded data.