# US 2022/0037022 A1 · Ensemble machine-learning models to detect respiratory syndromes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0037022 A1
- Document ID: US20220037022A1
- Status: Published application
- Kind: A1
- Application number: 17/393,113
- Filed: 2021-08-03
- Published: 2022-02-03
- Earliest claimed: 2020-08-03
- Assignees: The COVID Detection Foundation
- Applicants: Virutec, PBC
- Inventors: Amil Khanzada
- Web page: https://patentgrep.com/patent/US20220037022A1

## CPC

- G06N 20/20: Ensemble learning
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N 5/04: Inference or reasoning models
- G10L 25/66: for extracting parameters related to health condition (detecting or measuring for diagnostic purposes A61B5/00)
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- H03M 7/3059: {Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression}
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- H03M 7/3082: {Vector coding (for television signals, see H04N19/94)}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C)

## Abstract

Provided is a process including: obtaining, with one or more processors, a set of data comprising a plurality of patient records, selecting a subset of the plurality of parameters for inputs into a machine learning system, generating a classifier using the machine learning system based on the training data and the subset of the plurality of parameters for inputs; receiving, with one or more processors, patient record of a first user; performing an analysis, with one or more processors, to identify acoustic measures from a voice sample of the first user.

## Claims

### Claim 1

A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:
obtaining, with a computer system, a trained machine learning model configured to infer whether users have a respiratory illness based on both audio and an image captured by mobile computing devices of the users, wherein the trained machine learning model is trained by:
obtaining a training set comprising a plurality of training records, wherein:
each training record in the training set includes a plurality of parameters and corresponding values for a respective person;
each training record in the training set includes audio of the respective person's voice and an image of at least part of the respective person; and
each training record in the training set includes an indicator indicating whether or not the respective person has been diagnosed with a respiratory illness; and
training the machine learning model on the training set to infer whether users have the respiratory illness based on both the audio and images;
after obtaining the trained machine learning model, receiving, with the computer system, a first user record of a first user, the first user record comprising an audio file or stream of a voice of the first user and an image of at least part of the first user;
inferring, with the computer system, that the first user has the respiratory illness based on the audio file or stream of a voice of the first user and an image of at least part of the first user; and
storing, with the computer system, an indication that the first user has the respiratory illness in memory.

### Claim 2 (depends on 1)

The medium of claim 1, wherein the plurality of training records comprises at least two of the following:
textual questionnaire answers;
data indicative of breathing;
temporal data;
facial image;
fingertip video; or
biometric images of skin, stool, mucus, urine, or vomit.

### Claim 3 (depends on 1)

The medium of claim 1, wherein the plurality of training records comprises each of the following:
textual questionnaire answers;
data indicative of breathing;
temporal data;
facial image;
fingertip video; and
biometric images of skin, stool, mucus, urine, or vomit.

### Claim 4 (depends on 1)

The medium of claim 1, wherein the plurality of training records comprises fingertip video, wherein the machine learning model is trained to use the fingertip video to measure blood oxygen level and heartbeat rate as features upon which inference is based.

### Claim 5 (depends on 1)

The medium of claim 1, the operations further comprising:
steps for training the machine learning model.

### Claim 6 (depends on 1)

The medium of claim 1, wherein:
training the machine learning model comprises computing partial derivatives of parameters of the machine learning model with respect to an objective function and adjusting the parameters of the machine learning model in directions the partial derivatives indicate locally optimize the machine learning model.

### Claim 7 (depends on 1)

The medium of claim 1, wherein the machine learning model comprises:
at least two outputs, with a first output indicating COVID-19 infection and a second output indicating the stage of COVID-19 infection.

### Claim 8 (depends on 1)

The medium of claim 1, wherein:
the machine learning model comprises means for ensembling output of a plurality of sub-models.

### Claim 9 (depends on 1)

The medium of claim 1, wherein the operations further comprise:
configuring lossy compression of the audio file or stream of the person's voice to preserve non-human perceivable data that affects accuracy of the trained machine learning model.

### Claim 10 (depends on 1)

The medium of claim 1, wherein:
training the machine learning model is performed by a different set of computers of the computer system than a set of computers of the computer system that perform inferring the first user has the respiratory illness.

### Claim 11 (depends on 1)

The medium of claim 1, wherein:
inferring that the first user has the respiratory illness is performed by a smartphone of the first user that is part of the computer system.

### Claim 12 (depends on 1)

The medium of claim 1, wherein:
the trained machine learning model comprises an ensemble of at least three different machine learning algorithms having outputs combined with means for ensembling multiple sub-models.

### Claim 13 (depends on 1)

The medium of claim 1, the operations further comprising:
performing a pre-processing of an audio voice sample, before inputting to the trained machine learning model, to clean the voice sample and select segments of the voice sample to be input to the trained machine learning model.

### Claim 14 (depends on 1)

The medium of claim 1, wherein the operations further comprise:
extracting cepstral coefficients from the audio file or stream of the voice of the first user.

### Claim 15 (depends on 14)

The medium of claim 14, wherein extracting the cepstral coefficients comprises:
constructing a spectrogram from the audio file or stream of the voice of the first user;
calculating a log-power for each frame from the spectrogram;
applying a filter to a magnitude of the log-power;
performing a log-compression and transform to a cepstral domain of an output of the filter; and
forming a vector of cepstral coefficients for each frame.

### Claim 16 (depends on 1)

The medium of claim 1, wherein the operations comprise:
extracting mel-frequency cepstral coefficients derived from a power spectrum of audio of a voice sample of a second user.

### Claim 17 (depends on 1)

The medium of claim 1, wherein the trained machine learning model comprises a multi-layer feedforward neural network comprising at least two non-linear layers.

### Claim 18 (depends on 1)

The medium of claim 1, wherein the trained machine learning model comprises a double parallel feedforward neural network that takes a vector of mel-frequency cepstrum coefficients as an input.

### Claim 19 (depends on 1)

The medium of claim 1, wherein the operations comprise:
steps for inferring likelihood of COVID-19 infection.

### Claim 20

A method, comprising:
obtaining, with a computer system, a trained machine learning model configured to infer whether users have a respiratory illness based on both audio and an image captured by mobile computing devices of the users, wherein the trained machine learning model is trained by:
obtaining a training set comprising a plurality of training records, wherein:
each training record in the training set includes a plurality of parameters and corresponding values for a respective person;
each training record in the training set includes audio of the respective person's voice and an image of at least part of the respective person; and
each training record in the training set includes an indicator indicating whether or not the respective person has been diagnosed with a respiratory illness; and
training the machine learning model on the training set to infer whether users have the respiratory illness based on both the audio and images;
after obtaining the trained machine learning model, receiving, with the computer system, a first user record of a first user, the first user record comprising an audio file or stream of a voice of the first user and an image of at least part of the first user;
inferring, with the computer system, that the first user has the respiratory illness based on the audio file or stream of a voice of the first user and an image of at least part of the first user; and
storing, with the computer system, an indication that the first user has the respiratory illness in memory.
