# US 2022/0285015 A1 · Artificial intelligence for evaluating patient distress using facial emotion recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0285015 A1
- Document ID: US20220285015A1
- Status: Published application
- Kind: A1
- Application number: 17/630,027
- Filed: 2020-08-06
- Published: 2022-09-08
- Earliest claimed: 2019-08-06
- Assignees: University of Miami
- Applicants: University of Miami
- Inventors: Dipen J. PAREKH; Jonathan E. KATZ
- Web page: https://patentgrep.com/patent/US20220285015A1

## CPC

- G06F 3/012: {Head tracking input arrangements}
- G06F 18/24137: {Distances to cluster centroïds}
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/174: {Facial expression recognition}
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Artificial intelligence for evaluating patient distress using facial emotion recognition. In an embodiment, an artificial intelligence model is applied to facial image(s) of a patient to classify each facial image into one of a plurality of emotional states based on a facial expression in the facial image. A determination may be made as to whether or not to alert a healthcare provider based on the emotional state(s) into which the facial image(s) were classified. If a determination is made to alert a healthcare provider, a notification may be transmitted to the healthcare provider.

## Claims

### Claim 1

A method for monitoring patients in a medical facility comprising using at least one hardware processor to, for one or more patients in the medical facility:
receive one or more facial images of the patient, wherein each of the one or more facial images includes a face of the patient;
for each of the one or more facial images, apply an artificial-intelligence model to the facial image to classify the facial image into one of a plurality of emotional states based on a facial expression in the facial image;
determine whether or not to alert one or more recipients based on each emotional state into which each of the one or more facial images were classified; and,
when determining to alert the one or more recipients, transmit a notification to each of the one or more recipients.

### Claim 2 (depends on 1)

The method of claim 1, wherein the artificial-intelligence model comprises a machine-learning model trained on a plurality of facial images that have each been labeled with one of the plurality of emotional states.

### Claim 3 (depends on 2)

The method of claim 2, wherein the machine-learning model comprises a convolutional neural network.

### Claim 4 (depends on 1)

The method of claim 1, further comprising using the at least one hardware processor to store the emotional state into which the facial image was classified in memory.

### Claim 5 (depends on 1)

The method of claim 1, further comprising using a camera to capture the facial image, wherein the determination of whether or not to alert one or more recipients is made in real time with the capture of the facial image.

### Claim 6 (depends on 1)

The method of claim 1, wherein at least one of the plurality of emotional states indicates satisfaction, and at least one of the plurality of emotional states indicates dissatisfaction.

### Claim 7 (depends on 1)

The method of claim 1, wherein at least one of the plurality of emotional states indicates distress.

### Claim 8 (depends on 7)

The method of claim 7, wherein at least one of the plurality of emotional states indicates pain.

### Claim 9 (depends on 1)

The method of claim 1, wherein at least one of the plurality of emotional states indicates a risk of medical deterioration.

### Claim 10 (depends on 1)

The method of claim 1, wherein determining whether or not to alert one or more recipients comprises evaluating a plurality of emotional states output by the artificial-intelligence model for each of a plurality of received facial images of the patient.

### Claim 11 (depends on 10)

The method of claim 10, wherein evaluating the plurality of emotional states comprises applying an artificial-intelligence model to the plurality of emotional states to determine whether or not to alert the one or more recipients.

### Claim 12 (depends on 11)

The method of claim 11, wherein the artificial-intelligence model that is applied to the plurality of emotional states comprises a linear regression.

### Claim 13 (depends on 1)

The method of claim 1, wherein each of the one or more recipients is a healthcare provider.

### Claim 14 (depends on 13)

The method of claim 13, further comprising, for each of the one or more patients, capturing the facial image of the patient via a surveillance system in the medical facility.

### Claim 15 (depends on 14)

The method of claim 14, wherein the surveillance system comprises a camera directed at a hospital bed.

### Claim 16

A system comprising:
at least one hardware processor; and
one or more software modules that are configured to, when executed by the at least one hardware processor, for one or more persons,
receive one or more facial images of the person, wherein each of the one or more facial images includes a face of the person,
for each of the one or more facial images, apply an artificial-intelligence model to the facial image to classify the facial image into one of a plurality of emotional states based on a facial expression in the facial image,
determine whether or not to alert one or more recipients based on each emotional state into which each of the one or more facial images were classified, and,
when determining to alert the one or more recipients, transmit a notification to each of the one or more recipients.

### Claim 17

A non-transitory computer-readable medium having instructions stored therein, wherein the instructions, when executed by a processor, cause the processor to, for one or more persons:
receive one or more facial images of the person, wherein each of the one or more facial images includes a face of the person;
for each of the one or more facial images, apply an artificial-intelligence model to the facial image to classify the facial image into one of a plurality of emotional states based on a facial expression in the facial image;
determine whether or not to alert one or more recipients based on each emotional state into which each of the one or more facial images were classified; and,
when determining to alert the one or more recipients, transmit a notification to each of the one or more recipients.
