# US 11,908,128 B2 · Systems and methods to process images for skin analysis and to visualize skin analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,128 B2
- Document ID: US11908128B2
- Status: Granted
- Kind: B2
- Application number: 16/996,087
- Filed: 2020-08-18
- Published: 2021-01-14
- Granted: 2024-02-20
- Earliest claimed: 2019-07-10
- Assignees: L'oreal
- Applicants: L'Oreal
- Inventors: Ruowei Jiang; Irina Kezele; Zhi Yu; Sophie Seite; Frederic Antoinin Raymond Serge Flament; Parham Aarabi; Mathieu Perrot; Julien Despois
- Web page: https://patentgrep.com/patent/US11908128B2

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/441: {Skin evaluation, e.g. for skin disorder diagnosis}
- A61B 5/445: {Evaluating skin irritation or skin trauma, e.g. rash, eczema, wound, bed sore}
- A61B 5/6898: {Portable consumer electronic devices, e.g. music players, telephones, tablet computers}
- A61B 5/7267: {involving training the classification device}
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06Q 30/0631: {Recommending goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/161: {Detection; Localisation; Normalisation}
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/30: for calculating health indices; for individual health risk assessment
- A61B 5/7465: {Arrangements for interactive communication between patient and care services, e.g. by using a telephone network (telemetry of measured physiological signal A61B5/0002)}
- G06N 3/048: Activation functions
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30088: Skin; Dermal
- G06T 2207/30201: Face
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods process images to determine a skin condition severity analysis and to visualize a skin analysis such as using a deep neural network (e.g. a convolutional neural network) where a problem was formulated as a regression task with integer-only labels. Auxiliary classification tasks (for example, comprising gender and ethnicity predictions) are introduced to improve performance. Scoring and other image processing techniques may be used (e.g. in assoc. with the model) to visualize results such as highlighting the analyzed image. It is demonstrated that the visualization of results, which highlight skin condition affected areas, can also provide perspicuous explanations for the model. A plurality (k) of data augmentations may be made to a source image to yield k augmented images for processing. Activation masks (e.g. heatmaps) produced from processing the k augmented images are used to define a final map to visualize the skin analysis.

## Claims

### Claim 1

A skin diagnostic device comprising circuitry providing a processing unit coupled to a storage unit to configure the skin diagnostic device to provide:
a skin analysis unit to classify pixels of an image, that is a selfie face image from a user mobile device, using a deep neural network, which is already trained using a dataset of facial image data comprising selfie face images from user mobile devices, comprising a regressor and a classifier for image classification to generate the skin diagnosis for a skin condition;
wherein the deep neural network determines the skin diagnosis as an integer value on a scale classifying a severity of the skin condition over the image, and the deep neural network further provides an activation map to visualize the skin diagnosis in association with the image; and
wherein the skin diagnostic device is configured to:
perform a plurality (k) of data augmentations to the image to yield k augmented images for processing by the skin analysis unit;
process the k augmented images using the skin analysis unit to obtain k activation maps; and
define a final activation map for the image from the k activation maps to visualize the skin analysis.

### Claim 2 (depends on 1)

The diagnostic device of claim 1, wherein the processing unit and storage unit further configure the skin diagnostic device to provide a visualization unit to annotate the image to visualize the skin diagnosis using the final activation map.

### Claim 3 (depends on 1)

The diagnostic device of claim 1, wherein the processing unit and storage unit further configure the skin diagnostic device to provide a recommendation unit, responsive to the skin diagnosis, to recommend a product for the skin condition.

### Claim 4 (depends on 1)

The diagnostic device of claim 1, wherein the processing unit and storage unit further configure the skin diagnostic device to provide an e-commerce interface with which to purchase products for the skin condition.

### Claim 5 (depends on 1)

The diagnostic device of claim 1, wherein the skin condition comprises acne vulgaris and the scale classifying the severity of the skin condition over the image is a global acne severity scale (“GEA”).

### Claim 6 (depends on 1)

The diagnostic device of claim 1, wherein the deep neural network is configured to receive a normalized face image (x) as input and to output a vector y=ƒθ(x) which represents a probability distribution over all possible integer values on the scale and applies an activation function to determine the integer value of the skin diagnosis as a softmax expected value as follows:
E[ƒθ(x)]=Σi=ab(ƒθ(x)i*i)
where a, b are the lower and upper bound for the scale.

### Claim 7 (depends on 1)

The diagnostic device of claim 1, wherein:
the deep neural network comprises a shared encoder component for the regressor and classifier;
the deep neural network is further configured with one or both of an auxiliary ethnicity prediction component to predict ethnicity and an auxiliary gender prediction component to predict gender; and
the one or both of the auxiliary ethnicity prediction component and the auxiliary gender prediction component share the shared encoder component of the deep neural network.

### Claim 8 (depends on 1)

The diagnostic device of claim 1, wherein the deep neural network comprises a convolutional neural network (CNN) for image processing.

### Claim 9 (depends on 8)

The diagnostic device of claim 8, wherein the CNN comprises a residual network as an encoder having a global pooling operation before final fully connected layers configured to generate regressor and classifier outputs.

### Claim 10 (depends on 1)

The diagnostic device of claim 1, wherein the deep neural network is trained with a combined loss function combining a regressor loss function and a classifier loss function.

### Claim 11 (depends on 10)

The diagnostic device of claim 10, wherein the combined loss function conforms to an equation:
L=λmsemse+λcece+λgendergender+λethnicityethnicity (2)
where:
mse, ce, gender and ethnicity are respective mean square error (MSE), cross-entropy error (CE), gender prediction and ethnicity prediction functions; and
λmse, λce, λgender, and λethnicity are weighting factors.

### Claim 12 (depends on 1)

The diagnostic device of claim 1, wherein each of the k data augmentations comprises one augmentation randomly selected from an affine transformation or a horizontal flip operation.

### Claim 13 (depends on 1)

The diagnostic device of claim 1, wherein the deep neural network is configured to apply a Class Activation Mapping (CAM) technique to generate the activation maps.

### Claim 14 (depends on 1)

The diagnostic device of claim 1, wherein the diagnostic device comprises one of a smartphone, tablet, laptop, personal computer, or other computing device having or coupled to a camera.

### Claim 15 (depends on 3)

The diagnostic device of claim 3, wherein the recommendation unit provides a treatment product selector responsive to the skin diagnosis to obtain at least one of a product recommendation and a treatment plan recommendation.

### Claim 16 (depends on 1)

The diagnostic device of claim 1, wherein the skin condition is acne vulgaris.

### Claim 17

A computer implemented method of skin diagnosis comprising:
receiving an image, that is a selfie face image from a user mobile device;
performing a plurality (k) of data augmentations to the image to yield k augmented images for processing;
processing each of the k augmented images using a deep neural network to obtain k activation maps for a skin diagnosis, the deep neural network further configured to classify image pixels to determine the skin diagnosis for a skin condition, wherein the deep neural network, which is already trained using a dataset of facial image data comprising selfie face images from user mobile devices, is configured as a regressor and a classifier to determine the skin diagnosis as an integer value on a scale classifying a severity of the skin condition over the image, and is configured to provide an activation map to visualize the skin diagnosis;
defining a final activation map for the image from the k activation maps to visualize the skin diagnosis; and
visualizing the skin diagnosis in association with the image.

### Claim 18 (depends on 17)

The method of claim 17, wherein the deep neural network is further configured with auxiliary tasks to determine one or both of an ethnicity prediction and gender prediction.

### Claim 19 (depends on 17)

The method of claim 17, wherein the skin condition comprises acne vulgaris and the scale classifying the severity of the skin condition over the image is a global acne severity scale (“GEA”).

### Claim 20

A computer program product comprising a non-transitory storage device storing instruction that, when executed by a processor of a skin diagnostic device, cause the skin diagnostic device to:
receive an image, that is a selfie face image from a user mobile device;
perform a plurality (k) of data augmentations to the image to yield k augmented images for processing;
process each of the k augmented images using a deep neural network to obtain k activation maps for a skin diagnosis; and
define a final activation map for the image from the k activation maps to visualize the skin diagnosis; and
wherein the deep neural network is already trained using a dataset of facial image data comprising selfie face images from user mobile devices, and is configured:
to classify image pixels to determine the skin diagnosis for a skin condition,
to provide an activation map to visualize the skin diagnosis,
as a regressor and a classifier to determine the skin diagnosis, and
with auxiliary tasks to determine one or both of an ethnicity prediction and gender prediction.

### Claim 21 (depends on 1)

The diagnostic device of claim 1, wherein the final activation map is defined from the k activation maps using one or both of averaging and a threshold applied to eliminate values from the k activation maps.

### Claim 22 (depends on 1)

The diagnostic device of claim 1, wherein 2<=k<=20.