# US 2022/0273245 A1 · Image processing of streptococcal infection in pharyngitis subjects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0273245 A1
- Document ID: US20220273245A1
- Status: Published application
- Kind: A1
- Application number: 17/747,495
- Filed: 2022-05-18
- Published: 2022-09-01
- Earliest claimed: 2018-10-09
- Assignees: Light AI, Inc.
- Applicants: Light AI Inc.
- Inventors: Sarbjit Sarkaria; Steven Rebiffe; Udit Gupta; Mahendran Maliapen; Peter Whitehead
- Web page: https://patentgrep.com/patent/US20220273245A1

## CPC

- A61B 5/0084: {for introduction into the body, e.g. by catheters (A61B5/1459 takes precedence)}
- A61B 5/7267: {involving training the classification device}
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0035: {adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography}
- A61B 5/004: {adapted for image acquisition of a particular organ or body part (A61B5/0082 takes precedence; arrangements for optical scanning A61B5/0062)}
- A61B 5/0071: {by measuring fluorescence emission}
- A61B 5/01: Measuring temperature of body parts {; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue} (clinical contact thermometers G01K13/20)
- A61B 5/682: {Mouth, e.g., oral cavity; tongue; Lips; Teeth}
- A61B 5/7275: {Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor}
- A61B 5/742: {using visual displays (displays for heart-related electrical signals, e.g. ECG, A61B5/339)}
- A61K 9/0053: {Mouth and digestive tract, i.e. intraoral and peroral administration}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/443: {by matching or filtering}
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/82: using neural networks
- G06V 20/698: {Matching; Classification}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/67: for remote operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 5/0823: {Detecting or evaluating cough events}
- A61B 5/418: {lymph vessels, ducts or nodes}
- A61B 5/445: {Evaluating skin irritation or skin trauma, e.g. rash, eczema, wound, bed sore}
- A61B 2560/0431: Portable apparatus, e.g. comprising a handle or case
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- A61K 39/092: {Streptococcus}
- G06T 2207/10068: Endoscopic image
- G06T 2207/20084: Artificial neural networks [ANN]
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- Y02A 90/10: Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A61K: PREPARATIONS FOR MEDICAL, DENTAL OR TOILETRY PURPOSES (devices or methods specially adapted for bringing pharmaceutical products into particular physical or administering forms A61J3/00; chemical aspects of, or use of materials for deodorisation of air, for disinfection or sterilisation, or for bandages, dressings, absorbent pads or surgical articles A61L; soap compositions C11D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

A method for determining a disease state prediction, relating to a potential disease or medical condition of a subject, includes accessing a set of subject images, the subject images capturing a part of a subject's body, and accessing a set of clinical factors from the subject. The clinical factors are collected by a device or a medical practitioner substantially contemporaneously with the capture of the subject images. The subject images are inputted into an image model to generate disease metrics for disease prediction for the subject. The disease metrics generated by the image model and the clinical factors are inputted into a classifier to determine the disease state prediction, and the disease state prediction is returned.

## Claims

### Claim 1

A method comprising:
accessing a set of images of a subject, the images describing one or both of an oral cavity of the subject and a throat of the subject;
accessing a set of clinical factors from the subject, the clinical factors collected substantially contemporaneously with the capture of the images;
inputting one or more of the images into an image model to generate a pathogen presence prediction for the subject, wherein the image model was trained in part using training images, the training images including images of oral cavities and images of throats that include various bacterial pathogens, and the pathogen presence prediction describes in part whether any of the various bacterial pathogens are present in the subject;
inputting the pathogen presence prediction and the clinical factors into a machine learning model to determine a disease state prediction; and
returning the disease state prediction.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image model comprises:
a set of image parameter coefficients trained using a first set of training images and a first set of training labels, each corresponding to a first set of training subjects, the first set of training labels comprising:
a label indicating a presence of an abnormality, the abnormality selected from a group that includes a presence of a viral pathogen, and a presence of a bacterial pathogen,
a clear label indicating an absence of pathogens, and
a function relating one of the images and the image parameter coefficients to the pathogen presence prediction.

### Claim 3 (depends on 2)

The method of claim 2, wherein the machine learning model comprises:
a set of parameter coefficients trained using a set of training pathogen presence predictions, a set of training clinical factors, and a second set of training labels, each corresponding to a second set of training subjects,
the second set of training labels comprising:
a viral label indicating a presence of a viral pathogen,
a bacterial label indicating a presence of a bacterial pathogen, and
a clear subset indicating an absence of pathogens,
the set of training pathogen presence predictions generated by inputting a second set of training images corresponding to the second set of subjects into the image model;
the set of training clinical factors collected substantially contemporaneously with the capture of the second set of training images, and
a function relating the pathogen presence predictions, the clinical factors, and the parameter coefficients to the disease state prediction.

### Claim 4 (depends on 1)

The method of claim 1, wherein the set of training images was captured using an image capture device and the set of images was captured using the image capture device.

### Claim 5 (depends on 4)

The method of claim 4, wherein the set of training images and the set images each comprise:
a plurality of images captured under ambient light conditions;
a plurality of images captured under fluorescent light; and
a plurality of images captured under white light illumination.

### Claim 6 (depends on 1)

The method of claim 1, wherein the set of images is recorded with an image capture device comprising:
a light emitter configured to emit excitation light at a wavelength selected to elicit auto-fluorescence of a pathogen; and
a light sensor configured to detect light emissions or an absence of light emissions resulting from the auto-fluorescence of the pathogen.

### Claim 7 (depends on 6)

The method of claim 6, wherein the image capture device further comprises a display, the method further comprising:
displaying the disease state prediction on the display.

### Claim 8 (depends on 1)

The method of claim 1, wherein the set of subject images is recorded by a mobile phone device.

### Claim 9 (depends on 8)

The method of claim 8, wherein the disease state prediction is displayed on the mobile phone device.

### Claim 10 (depends on 1)

The method of claim 1, wherein the set images of the subject comprises at least one of:
at least one image of the throat of the subject captured using a blue light emitter;
a throat image captured using a white light emitter; and
a throat image captured using ambient light.

### Claim 11 (depends on 1)

The method of claim 1, wherein the images captures data regarding multiple wavelengths of light.

### Claim 12 (depends on 1)

The method of claim 1, wherein at least one of the images captures infrared light image data.

### Claim 13 (depends on 1)

The method of claim 1, wherein at least one of images is pre-processed before being input into the image model, the pre-processing comprising at least one from the group consisting of:
uniform aspect ratio correction,
rescaling,
normalization,
object detection,
segmentation,
cropping,
dimensionality reduction,
dimensionality increment,
brightness adjustment,
image shifting,
image flipping,
zoom in or out,
image rotation,
image quality filtering, and
image pixel correction.

### Claim 14 (depends on 1)

The method of claim 1, wherein the image model is a machine learning model.

### Claim 15 (depends on 1)

The method of claim 1, wherein the classifier is trained using one of: linear regression, logistic regression, multinomial regression, and elastic net regression.

### Claim 16 (depends on 1)

The method of claim 1, wherein the classifier is one of a random forest classifier, a gradient boosted classifier, a support vector machine classifier, and a Naive Bayes classifier.

### Claim 17 (depends on 1)

The method of claim 1, wherein the pathogen presence prediction comprises at least one of:
a probability of a presence of a viral pathogen,
a probability of a presence of a bacterial pathogen, and
a probability of an absence of a pathogen.

### Claim 18 (depends on 1)

The method of claim 1, wherein the pathogen presence prediction comprises at least one of:
a probability of a presence of exudate,
a probability of a presence of petechiae,
a probability of a presence of swollen tonsils, and
a probability of a presence of a swollen uvula.

### Claim 19 (depends on 1)

The method of claim 1, the disease state prediction comprises at least one of:
a probability of viral pathogen infection,
a probability of bacterial pathogen infection, and
a probability of no pathogen infection.

### Claim 20 (depends on 1)

The method of claim 1, wherein the set of clinical factors comprises at least one from the group consisting of:
age,
a presence or absence of swollen lymph nodes,
subject temperature,
a presence or absence of a fever, and
a presence or absence of a cough.

### Claim 21 (depends on 1)

The method of claim 1, wherein the set of clinical factors comprises at least one from the group consisting of:
age,
a presence or absence of swollen lymph nodes,
subject temperature,
a presence or absence of fever,
a presence or absence of a cough,
a presence or absence of a runny nose,
a presence or absence of a headache,
a presence or absence of body aches,
a presence or absence of vomiting,
a presence or absence of diarrhea,
a presence or absence of fatigue,
a presence or absence of chills, and
a duration of pharyngitis.

### Claim 22

A computer system comprising a computer processor and a memory, the memory storing computer program instructions that when executed by the computer processor cause the processor to:
access a set of images of a subject, the images describing one or both of an oral cavity of the subject and a throat of the subject;
access a set of clinical factors from the subject, the clinical factors collected substantially contemporaneously with the capture of the images;
input one or more of the images into an image model to generate a pathogen presence prediction for the subject, wherein the image model was trained in part using training images, the training images including images of oral cavities and images of throats that include various bacterial pathogens, and the pathogen presence prediction describes in part whether any of the various bacterial pathogens are present in the subject;
input the pathogen presence prediction and the clinical factors into a machine learning model to determine a disease state prediction; and
return the disease state prediction.

### Claim 23

A non-transitory computer readable storage medium comprising computer program instructions that when executed by a computer processor cause the processor to:
access a set of images of a subject, the images describing one or both of an oral cavity of the subject and a throat of the subject;
access a set of clinical factors from the subject, the clinical factors collected substantially contemporaneously with the capture of the images;
input the images into an image model to generate a pathogen presence prediction for the subject, wherein the image model was trained in part using training images that include various viral and/or bacterial pathogens, and the pathogen presence prediction describes in part whether any of the various viral and/or bacterial pathogens are present in the subject;
input the pathogen presence prediction and the clinical factors into a machine learning model to determine a disease state prediction; and
return the disease state prediction.

### Claim 24

A method comprising:
accessing a set of subject images, the subject images capturing a part of a subject's body;
accessing a set of clinical factors from the subject, the clinical factors collected substantially contemporaneously with the capture of the subject images;
inputting the subject images into an image model to generate disease metrics for disease prediction for the subject, wherein the image model was trained in part using training images that include various viral and/or bacterial pathogens, and the disease metrics describe in part whether any of the various viral and/or bacterial pathogens are present in the subject;
inputting the disease metrics and the clinical factors into a classifier to determine a disease state prediction, the disease state prediction relating to a disease or medical condition; and
returning the disease state prediction.

### Claim 25 (depends on 24)

The method of claim 24, wherein the disease metrics comprise:
feature metrics corresponding to identified features in the subject image, and
infection metrics corresponding to a presence of a bacterial or viral infection in the part of the subject's body.

### Claim 26 (depends on 24)

The method of claim 24, wherein the image model comprises:
a set of image parameter coefficients trained using a first set of training subject images and a first set of training labels, each corresponding to a first set of training subjects, the first set of training labels comprising:
a label indicating a presence of an abnormality, the abnormality selected from a group that includes a viral pathogen and a bacterial pathogen, and
a clear label indicating an absence of pathogens; and
a function relating one of the subject images and the image parameter coefficients to the disease metrics.

### Claim 27 (depends on 24)

The method of claim 24, wherein the machine learning model comprises:
a set of classifier parameter coefficients trained using a set of training disease metrics, a set of training clinical factors, and a second set of training labels, each corresponding to a second set of training subjects,
the second set of training labels comprising:
a label indicating a presence of an abnormality, the abnormality selected from a group that includes a viral pathogen and a bacterial pathogen, and
a clear subset indicating an absence of pathogens,
the set of training disease metrics generated by inputting a second set of training subject images corresponding to the second set of subjects into the image model;
the set of training clinical factors collected by a device or a medical practitioner substantially contemporaneously with the capture of the second set of training subject images, and
a function relating the disease metrics, the clinical factors, and the classifier parameter coefficients to the disease state prediction.
