# US 10,311,334 B1 · Learning to process images depicting faces without leveraging sensitive attributes in deep learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,334 B1 - Document ID: US10311334B1 - Status: Granted - Kind: B1 - Application number: 16/213,059 - Filed: 2018-12-07 - Granted: 2019-06-04 - Earliest claimed: 2018-12-07 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Omar Florez Choque; Erik Mueller - Web page: https://patentgrep.com/patent/US10311334B1 ## CPC - G06V 10/82: using neural networks - G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/7747: {Organisation of the process, e.g. bagging or boosting} - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/174: {Facial expression recognition} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems, methods, and articles of manufacture to generate, by a neural network of a variational autoencoder, a latent vector for a first input image, generate, by the neural network of the variational autoencoder, a first reconstructed image by sampling the latent vector for the first input image, determine a reconstruction loss incurred in generating the first reconstructed image based at least in part on: (i) a difference of the first input image and the first reconstructed image, and (ii) a master model trained to detect a sensitive attribute in images, determine a total loss based at least in part on the reconstruction loss and a classification loss, and optimize a plurality of weights of the neural network of the variational autoencoder based on a backpropagation operation and the determined total loss, the optimized neural network trained to not consider the sensitive attribute in images. ## Claims ### Claim 1 A system, comprising: a processor circuit; and a memory storing instructions which when executed by the processor circuit cause the processor circuit to: generate, by a neural network of a variational autoencoder, a latent vector for a first input image; generate, by the neural network of the variational autoencoder, a first reconstructed image by sampling the latent vector for the first input image; determine a reconstruction loss incurred in generating the first reconstructed image based at least in part on: (i) a difference of the first input image and the first reconstructed image, and (ii) a master model trained to detect a sensitive attribute in images; determine a total loss based at least in part on the reconstruction loss and a classification loss; and optimize a plurality of weights of the neural network of the variational autoencoder based on a backpropagation operation and the determined total loss, the optimization to train the neural network to not consider the sensitive attribute when processing images. ### Claim 2 (depends on 1) The system of claim 1, the memory storing instructions which when executed by the processor circuit cause the processor circuit to: receive a second input image; generate, by the neural network of the variational autoencoder, a latent vector for the second input image; and generate, by the neural network of the variational autoencoder, a second reconstructed image by sampling the latent vector for the second input image, the master model not able to detect the sensitive attribute in the second reconstructed image. ### Claim 3 (depends on 2) The system of claim 2, the memory storing instructions which when executed by the processor circuit cause the processor circuit to: perform an image processing operation on the second reconstructed image, the image processing operation comprising one or more of: (i) determining whether a face is depicted in the second reconstructed image, and (ii) determining whether the face depicted in the second reconstructed image is expressing one or more emotions. ### Claim 4 (depends on 1) The system of claim 1, the memory storing instructions to determine the classification loss for the first reconstructed image which when executed by the processor circuit cause the processor circuit to: compute a difference of a predetermined latent vector for the first input image and an output of a convolutional neural network receiving the first reconstructed image as input; and compute a square of the computed difference. ### Claim 5 (depends on 4) The system of claim 4, the memory storing instructions which when executed by the processor circuit cause the processor circuit to: determine the total loss for the first reconstructed image based on a sum of the classification loss and the reconstruction loss. ### Claim 6 (depends on 5) The system of claim 5, the memory storing instructions to determine the reconstruction loss which when executed by the processor circuit cause the processor circuit to: compute a first difference between the first input image and the first reconstructed image; compute a square of the first difference; compute a second difference between a first output of the master model and a second output of the master model, the first output based on the first input image, the second output based on the first reconstructed image; compute a square of the second difference; and compute a difference of the square of the first difference and the square of the second difference. ### Claim 7 (depends on 1) The system of claim 1, wherein the sensitive attribute comprises one or more of: (i) a gender, (ii) a race, (iii) an age, (iv) a skin tone, and (v) a religion of a person depicted in images, wherein the master model comprises a neural network. ### Claim 8 A non-transitory computer-readable storage medium storing instructions that when executed by a processor of a computing device, cause the processor to: generate, by a neural network of a variational autoencoder, a latent vector for a first input image; generate, by the neural network of the variational autoencoder, a first reconstructed image by sampling the latent vector for the first input image; determine a reconstruction loss incurred in generating the first reconstructed image based at least in part on: (i) a difference of the first input image and the first reconstructed image, and (ii) a master model trained to detect a sensitive attribute in images; determine a total loss based at least in part on the reconstruction loss and a classification loss; and optimize a plurality of weights of the neural network of the variational autoencoder based on a backpropagation operation and the determined total loss, the optimization to train the neural network to not consider the sensitive attribute when processing images. ### Claim 9 (depends on 8) The non-transitory computer-readable storage medium of claim 8, further storing instructions that when executed by the processor cause the processor to: receive a second input image; generate, by the neural network of the variational autoencoder, a latent vector for the second input image; and generate, by the neural network of the variational autoencoder, a second reconstructed image by sampling the latent vector for the second input image, the master model not able to detect the sensitive attribute in the second reconstructed image. ### Claim 10 (depends on 9) The non-transitory computer-readable storage medium of claim 9, further storing instructions that when executed by the processor cause the processor to: perform an image processing operation on the second reconstructed image, the image processing operation comprising one or more of: (i) determining whether a face is depicted in the second reconstructed image, and (ii) determining whether the face depicted in the second reconstructed image is expressing one or more emotions. ### Claim 11 (depends on 8) The non-transitory computer-readable storage medium of claim 8, further storing instructions that when executed by the processor cause the processor to: compute a difference of a predetermined latent vector for the first input image and an output of a convolutional neural network receiving the first reconstructed image as input; and compute a square of the computed difference to determine the classification loss for the first reconstructed image. ### Claim 12 (depends on 11) The non-transitory computer-readable storage medium of claim 11, further storing instructions that when executed by the processor cause the processor to: determine the total loss for the first reconstructed image based on a sum of the classification loss and the reconstruction loss. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, further storing instructions that when executed by the processor cause the processor to: compute a first difference between the first input image and the first reconstructed image; compute a square of the first difference; compute a second difference between a first output of the master model and a second output of the master model, the first output based on the first input image, the second output based on the first reconstructed image; compute a square of the second difference; and compute, as the reconstruction loss, a difference of the square of the first difference and the square of the second difference. ### Claim 14 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein the sensitive attribute comprises one or more of: (i) a gender, (ii) a race, (iii) an age, (iv) a skin tone, and (v) a religion of a person depicted in images, wherein the master model comprises a neural network. ### Claim 15 A method, comprising: generating, by a neural network of a variational autoencoder, a latent vector for a first input image; generating, by the neural network of the variational autoencoder, a first reconstructed image by sampling the latent vector for the first input image; determining a reconstruction loss incurred in generating the first reconstructed image based at least in part on: (i) a difference of the first input image and the first reconstructed image, and (ii) a master model trained to detect a sensitive attribute in images; determining a total loss based at least in part on the reconstruction loss and a classification loss; and optimizing a plurality of weights of the neural network of the variational autoencoder based on a backpropagation operation and the determined total loss, the optimization to train the neural network to not consider the sensitive attribute when processing images. ### Claim 16 (depends on 15) The method of claim 15, further comprising: receiving a second input image; generating, by the neural network of the variational autoencoder, a latent vector for the second input image; and generating, by the neural network of the variational autoencoder, a second reconstructed image by sampling the latent vector for the second input image, the master model not able to detect the sensitive attribute in the second reconstructed image. ### Claim 17 (depends on 16) The method of claim 16, further comprising: performing an image processing operation on the second reconstructed image, the image processing operation comprising one or more of: (i) determining whether a face is depicted in the second reconstructed image, and (ii) determining whether the face depicted in the second reconstructed image is expressing one or more emotions. ### Claim 18 (depends on 15) The method of claim 15, the method further comprising: computing a difference of a predetermined latent vector for the first input image and an output of a convolutional neural network receiving the first reconstructed image as input; and computing a square of the computed difference to determine the classification loss for the first reconstructed image. ### Claim 19 (depends on 18) The method of claim 18, further comprising: determining the total loss for the first reconstructed image based on a sum of the classification loss and the reconstruction loss. ### Claim 20 (depends on 15) The method of claim 15, wherein the sensitive attribute comprises one or more of: (i) a gender, (ii) a race, (iii) an age, (iv) a skin tone, and (v) a religion of a person depicted in images, wherein the master model comprises a neural network, the method further comprising: computing a first difference between the first input image and the first reconstructed image; computing a square of the first difference; computing a second difference between a first output of the master model and a second output of the master model, the first output based on the first input image, the second output based on the first reconstructed image; computing a square of the second difference; and computing, as the reconstruction loss, a difference of the square of the first difference and the square of the second difference.