# US 12,014,828 B2 · Using a set of machine learning diagnostic models to determine a diagnosis based on a skin tone of a patient

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,014,828 B2
- Document ID: US12014828B2
- Status: Granted
- Kind: B2
- Application number: 17/619,173
- Filed: 2020-06-15
- Published: 2022-09-29
- Granted: 2024-06-18
- Earliest claimed: 2019-06-18
- Assignees: Digital Diagnostics Inc.
- Applicants: Digital Diagnostics Inc.
- Inventors: Elliot Swart; Elektra Efstratiou Alivisatos; Joseph Ferrante; Elizabeth Asai
- Web page: https://patentgrep.com/patent/US12014828B2

## CPC

- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/1032: {Determining colour of tissue for diagnostic purposes}
- A61B 5/441: {Skin evaluation, e.g. for skin disorder diagnosis}
- A61B 5/444: {Evaluating skin marks, e.g. mole, nevi, tumour, scar}
- A61B 5/6898: {Portable consumer electronic devices, e.g. music players, telephones, tablet computers}
- A61B 5/7267: {involving training the classification device}
- A61B 5/7275: {Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor}
- G06F 18/24765: {Rule-based classification}
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0014: {using an image reference approach}
- G06T 7/90: Determination of colour characteristics
- G06V 10/56: relating to colour
- G06V 10/72: Data preparation, e.g. statistical preprocessing of image or video features
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/67: for remote operation
- A61B 2576/02: specially adapted for a particular organ or body part
- G06T 2207/10024: Color image
- G06T 2207/20008: Globally adaptive
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30088: Skin; Dermal
- G06T 2207/30168: Image quality inspection
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods are disclosed herein for determining a diagnosis based on a base skin tone of a patient. In an embodiment, the system receives a base skin tone image of a patient, generates a calibrated base skin tone image by calibrating the base skin tone image using a reference calibration profile, and determines a base skin tone of the patient based on the calibrated base skin tone image. The system receives a concern image of a portion of the patient's skin, and selects a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the base skin tone of the patient, each of the sets of candidate machine learning diagnostic models trained to receive the concern image and output a diagnosis of a condition of the patient.

## Claims

### Claim 1

A method for determining a diagnosis based on a base skin tone of a patient, the method comprising:
receiving a base skin tone image of a patient;
generating a calibrated base skin tone image by calibrating the base skin tone image using a reference calibration profile;
determining a base skin tone of the patient based on the calibrated base skin tone image;
receiving a concern image of a portion of the patient's skin;
selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the base skin tone of the patient, each of the sets of candidate machine learning diagnostic models trained to receive the concern image and output a diagnosis of a condition of the patient;
inputting the base skin tone and the concern image into at least one classifier of the selected set of machine learning diagnostic models; and
receiving, as output from the at least one classifier, information from which the diagnosis is derived.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the calibrated skin tone image is performed responsive to determining that the base skin tone image satisfies a quality criterion.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the base skin tone of the patient based on the calibrated image comprises:
determining a numerical representation of each pixel of the calibrated base skin tone image;
generating an aggregate representation by performing a statistical operation based on each numerical representation;
identifying a point in a color space corresponding to the aggregate representation; and
mapping the point in the color space into a discrete value in a classification system.

### Claim 4 (depends on 1)

The method of claim 1, wherein the method further comprises generating a plurality of calibrated base skin tone images, the plurality comprising the calibrated base skin tone image, and wherein determining the base skin tone of the patient based on the calibrated image further comprises:
inputting each of the plurality of calibrated base skin tone images into a measurement classifier; and
receiving as output from the measurement classifier the base skin tone.

### Claim 5 (depends on 1)

The method of claim 1, wherein the method further comprises:
generating an adapted imaging profile based on the reference calibration profile and the base skin tone; and
calibrating the concern image using the adapted imaging profile.

### Claim 6 (depends on 1)

The method of claim 1, wherein the method further comprises determining a classification that corresponds to the base skin tone, wherein selecting the set of machine learning diagnostic models comprises selecting a diagnosis classifier that corresponds to the classification, and wherein the method further comprises:
inputting the concern image into the selected diagnosis classifier; and
receiving, as output from the selected diagnosis classifier, information from which the diagnosis is derived.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the method further comprises:
inputting the concern image into at least one classifier of the selected set of machine learning diagnostic models;
receiving, as output from the at least one classifier, information from which the diagnosis is derived;
modifying the information based on the base skin tone; and
determining the diagnosis based on the modified information.

### Claim 8 (depends on 1)

The method of claim 1, wherein the method further comprises:
determining whether the concern image is diagnosable based on the base skin tone; and
responsive to determining that the concern image is not diagnosable, outputting an indication that the concern image is not diagnosable.

### Claim 9 (depends on 1)

The method of claim 1, wherein the diagnosis of the condition comprises a probability that the patient has the condition.

### Claim 10

A computer program product for determining a diagnosis based on a base skin tone of a patient, the computer program product comprising a non-transitory computer-readable storage medium containing computer program code for:
receiving a base skin tone image of a patient;
generating a calibrated base skin tone image by calibrating the base skin tone image using a reference calibration profile;
determining a base skin tone of the patient based on the calibrated base skin tone image;
receiving a concern image of a portion of the patient's skin;
selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the base skin tone of the patient, each of the sets of candidate machine learning diagnostic models trained to receive the concern image and output a diagnosis of a condition of the patient;
inputting the base skin tone and the concern image into at least one classifier of the selected set of machine learning diagnostic models; and
receiving, as output from the at least one classifier, information from which the diagnosis is derived.

### Claim 11 (depends on 10)

The computer program product of claim 10, wherein generating the calibrated skin tone image is performed responsive to determining that the base skin tone image satisfies a quality criterion.

### Claim 12 (depends on 10)

The computer program product of claim 10, wherein determining the base skin tone of the patient based on the calibrated image comprises:
determining a numerical representation of each pixel of the calibrated base skin tone image;
generating an aggregate representation by performing a statistical operation based on each numerical representation;
identifying a point in a color space corresponding to the aggregate representation; and
mapping the point in the color space into a discrete value in a classification system.

### Claim 13 (depends on 10)

The computer program product of claim 10, wherein the computer code is further for generating a plurality of calibrated base skin tone images, the plurality comprising the calibrated base skin tone image, and wherein determining the base skin tone of the patient based on the calibrated image further comprises:
inputting each of the plurality of calibrated base skin tone images into a measurement classifier; and
receiving as output from the measurement classifier the base skin tone.

### Claim 14 (depends on 10)

The computer program product of claim 10, wherein the computer code is further for determining a classification that corresponds to the base skin tone, wherein selecting the set of machine learning diagnostic models comprises selecting a diagnosis classifier that corresponds to the classification, and wherein the computer code is further for:
inputting the concern image into the selected diagnosis classifier; and
receiving, as output from the selected diagnosis classifier, information from which the diagnosis is derived.

### Claim 15 (depends on 10)

The computer program product of claim 10, wherein the computer code is further for:
inputting the concern image into at least one classifier of the selected set of machine learning diagnostic models;
receiving, as output from the at least one classifier, information from which the diagnosis is derived;
modifying the information based on the base skin tone; and
determining the diagnosis based on the modified information.

### Claim 16 (depends on 10)

The computer program product of claim 10, wherein the diagnosis of the condition comprises a probability that the patient has the condition.

### Claim 17

A method for determining a diagnosis of a patient, the method comprising:
receiving biometric data of at least a portion of a patient;
inputting the biometric data into a classifier trained to obtain an attribute of the patient;
receiving, as output from the classifier, the attribute of the patient; and
selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the attribute of the patient, each of the sets of candidate machine learning diagnostic models trained to receive the attribute and output a diagnosis of a condition of the patient, wherein each of the sets of candidate machine learning diagnostic models are trained to receive a concern image depicting an organ of the patient in addition to the attribute, and to output the diagnosis of the condition of the patient based on both the attribute and the concern image.