# US 2023/0419549 A1 · Systems and methods for determining and using a multidimensional measure of apparent skin color

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419549 A1
- Document ID: US20230419549A1
- Status: Published application
- Kind: A1
- Application number: 18/333,729
- Filed: 2023-06-13
- Published: 2023-12-28
- Earliest claimed: 2022-06-24
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: William Eric Thong; Przemyslaw Kamil Joniak; Alice Xiang
- Web page: https://patentgrep.com/patent/US20230419549A1

## CPC

- G06T 7/90: Determination of colour characteristics
- G06V 40/162: {using pixel segmentation or colour matching}
- G06T 7/11: Region-based segmentation
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/172: {Classification, e.g. identification}
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods can compute a multidimensional score of apparent skin color to capture skin color variation and provide a comprehensive representation of its constitutive complexity. The multidimensional apparent skin color score can be used to reveal biases related to skin color in image datasets and computer vision models. The multidimensional skin color score can focus on the perceptual lightless, L*, as a measure of skin tone ranging from light to dark, and the hue angle h*, as a measure of skin hue ranging from red to yellow. The multidimensional skin color scale offers a more representative assessment to surface socially relevant biases due to skin color effects in computer vision. This could help to (i) enhance the diversity in the data collection process, by encouraging specifications to better represent skin color variability; and (ii) improve the identification of dataset and model biases in fairness benchmarking, by highlighting their limitations.

## Claims

### Claim 1

A computer-implemented method for determining an apparent skin color of a face image, comprising:
receiving an input image;
segmenting skin pixels in the input image;
extracting a perceptual lightness, L*, value, a green-red opponent color value, a*, and a blue-yellow opponent color value in each segmented skin pixel;
applying a clustering algorithm to cluster the segmented skin pixels;
computing a histogram of a distribution obtained from the clustering algorithm; and
assigning a scalar value for L* and for a hue angle, h*, based on a mode of the histogram, for each cluster.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising determining a final scalar score for the input image by averaging the scalar values of each cluster, normalized by a pixel size of each cluster.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising excluding clusters that have an L* value below a predetermined minimum.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein:
the clustering algorithm provides five clusters; and
out of the five clusters, three clusters with the greatest L* value are retained to compute the L* and h* scalar values.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein a number of bins of the histogram is determined by a Sturges formula.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the segmentation of the skin pixels is performed with a skin segmentation machine learning model.

### Claim 7

A computer-implemented method for determining bias in a face image dataset, comprising:
determining an apparent skin color of each face image in the face image dataset by:
receiving an input image having the face image;
segmenting skin pixels in the input image;
extracting a perceptual lightness, L*, value, a green-red opponent color value, a*, and a blue-yellow opponent color value in each segmented skin pixel;
applying a clustering algorithm to cluster the segmented skin pixels;
computing a histogram of a distribution obtained from the clustering algorithm;
assigning a scalar value for L* and for a hue angle, h*, based on a mode of the histogram, for each cluster; and
determining a final scalar score for the input image by averaging the scalar values of each cluster, normalized by a pixel size of each cluster;
determining the bias in the face image dataset with respect to light and dark skin tones by calculating a number of face images having the L* scalar value greater than and less than a first predetermined threshold; and
determining the bias in the face image dataset with respect to skin hue by calculating a number of face images having the h* scalar value greater than and less than a second predetermined threshold.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising excluding clusters that have an L* value below a predetermined minimum.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein:
the clustering algorithm provides five clusters; and
out of the five clusters, three clusters with the greatest L* value are retained to compute the L* and h* scalar values.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, wherein a number of bins of the histogram is determined by a Sturges formula.

### Claim 11 (depends on 7)

The computer-implemented method of claim 7, wherein the segmentation of the skin pixels is performed with a skin segmentation machine learning model.

### Claim 12 (depends on 7)

The computer-implemented method of claim 7, further comprising:
generating images with a generative adversarial network and a diffusion model; and
determining the apparent skin color of each face image in the generated images; and
determining whether the determined bias is present in the generated images.

### Claim 13

A computer-implemented method for determining bias in a computer vision-based model, comprising:
determining an apparent skin color of each face image in the face image dataset by:
receiving an input image having the face image;
segmenting skin pixels in the input image;
extracting a perceptual lightness, L*, value, a green-red opponent color value, a*, and a blue-yellow opponent color value in each segmented skin pixel;
applying a clustering algorithm to cluster the segmented skin pixels;
computing a histogram of a distribution obtained from the clustering algorithm;
assigning a scalar value for L* and for a hue angle, h*, based on a mode of the histogram, for each cluster; and
determining a final scalar score for the input image by averaging the scalar values of each cluster, normalized by a pixel size of each cluster;
using the face image dataset in a machine learning model; and
determining whether an output of the machine learning model includes a bias based on at least one of the perceptual lightness value or the hue angle value.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein the machine learning model is a face cropping model that selects one face image from a group of two face images, the method further comprising:
measuring a probability of the one face image to be selected by the face cropping model;
comparing the perceptual lightness value and the hue angle value of the two face images; and
determining a bias in at least one of the perceptual lightness value or the hue angle value when the face cropping model selects the one face image.

### Claim 15 (depends on 13)

The computer-implemented method of claim 13, wherein the machine learning model is a face recognition model that determines whether two face images are from the same person, the method further comprising:
determining a percentage of correct determinations that the face recognition model correctly determine whether the two face images are from the same person; and
determining bias between the percentage of correct determines and at least one of the perceptual lightness value or the hue angle value.

### Claim 16 (depends on 13)

The computer-implemented method of claim 13, wherein the machine learning model is an attribute prediction model that determines whether the face image includes a selected attribute, the method further comprising:
manipulating the face image dataset by changing a skin tone and a skin hue of each image therein; and
observing a causal effect of the manipulating on a performance of correct prediction of the selected attribute with the attribute prediction model.

### Claim 17 (depends on 13)

The computer-implemented method of claim 13, further comprising excluding clusters that have an L* value below a predetermined minimum.

### Claim 18 (depends on 13)

The computer-implemented method of claim 13, wherein:
the clustering algorithm provides five clusters; and
out of the five clusters, three clusters with the greatest L* value are retained to compute the L* and h* scalar values.
