# US 2021/0241603 A1 · Hand sanitation compliance enforcement systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241603 A1
- Document ID: US20210241603A1
- Status: Published application
- Kind: A1
- Application number: 17/236,202
- Filed: 2021-04-21
- Published: 2021-08-05
- Earliest claimed: 2019-10-09
- Assignees: Visualq
- Applicants: Visualq
- Inventors: John Caleb Harman; John M. Abdelsayed
- Web page: https://patentgrep.com/patent/US20210241603A1

## CPC

- G06Q 30/0261: {based on user location}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06K 7/10297: {arrangements for handling protocols designed for non-contact record carriers such as RFIDs NFCs, e.g. ISO/IEC 14443 and 18092 (cryptographic protocols H04L9/00; network security protocols H04L63/00; real-time communication protocols in data switching networks H04L65/00; network protocols for data switching network services H04L67/00)}
- G06K 7/10415: {the interrogation device being fixed in its position, such as an access control device for reading wireless access cards, or a wireless ATM (banking machines in general G07F19/00)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06Q 10/10: Office automation; Time management
- G06Q 30/018: Certifying business or products
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G08B 5/36: using visible light sources
- G08B 21/245: {Reminder of hygiene compliance policies, e.g. of washing hands}
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 40/20: for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
- H04N 7/183: {for receiving images from a single remote source}
- G16H 30/40: for processing medical images, e.g. editing
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An electronic processor is operatively connected with at least one video camera, or additionally or alternatively a non-camera-based sensor, and at least one annunciator to process real time video acquired by the at least one video camera in real time to perform a hand hygiene compliance enforcement method for encouraging use of a hand sanitation station. The real time video is analyzed to determine an occupancy time interval over which a sanitation zone imaged by the real time video is occupied. Whether a hand sanitation event occurs is determined based on images of a wash station zone extracted from the real time video acquired during the occupancy time interval. The at least one annunciator is controlled to annunciate a hand hygiene compliance status based on the determination of whether a hand sanitation event occurs.

## Claims

### Claim 1

A hand hygiene compliance enforcement device comprising:
at least one video camera;
at least one annunciator;
an electronic processor; and
a non-transitory storage medium storing instructions that are readable and executable by the electronic processor, wherein the instructions, when executed by the electronic processor, cause the electronic processor to:
process real time video acquired by the at least one video camera in real time,
analyze, by execution of occupancy detection instructions, the real time video to determine an occupancy time interval over which a sanitation zone imaged by the real time video is occupied,
determine, by execution of sanitation detection instructions, whether a hand sanitation event occurs based on images of a wash station zone as extracted from the real time video acquired during the occupancy time interval, and
control the at least one annunciator to annunciate a hand hygiene compliance status based on the determination of whether a hand sanitation event occurs.

### Claim 2 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the electronic processor is configured to execute the sanitation detection instructions to cause the electronic processor to:
apply, for each respective frame of successive frames of the real time video acquired during the occupancy time interval, a machine learning (ML) classifier to one or more images of the wash station zone as extracted from the respective frame to generate a confidence value indicative of likelihood that the one or more images of the wash station zone depicts a hand sanitation event, and
determine whether a hand sanitation event occurs in the wash station zone based on the confidence values for the successive frames,
wherein the ML classifier is pre-trained to generate the confidence value using (1) training images of the wash station zone or a standard wash station zone that depicts a hand sanitation event, and (2) training images of the wash station zone or the standard wash station zone that do not depict a hand sanitation event.

### Claim 3 (depends on 2)

The hand hygiene compliance enforcement device of claim 2, wherein application of the ML classifier comprises:
converting an image of the wash station zone as extracted from a frame of the successive frames to a tensor of channel arrays, wherein each channel array represents a color channel of the image of the wash station zone; and
applying the ML classifier to the tensor of channel arrays.

### Claim 4 (depends on 1)

The hand hygiene compliance enforcement device of claim 1,
wherein the images of the wash station zone, as extracted from the real time video acquired during the occupancy time interval, have pixel dimensions length (L) by width (W) where L is 100 pixels or less and W is 100 pixels or less, and
wherein extraction of the images of the wash station zone from the real time video acquired during the occupancy time interval comprises pixel resampling, the pixel dimensions L by W being determined after the pixel resampling.

### Claim 5 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the electronic processor is configured to execute the sanitation detection instructions to cause the electronic processor to:
determine whether a hand sanitation event occurs in the wash station zone by detecting a color of a hand sanitation fluid in the images of the wash station zone as extracted from the real time video acquired during the occupancy time interval.

### Claim 6 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the electronic processor is configured to execute the occupancy detection instructions to cause the electronic processor to perform at least one of:
a) extracting, for one or more successive frames of the real time video, an extracted image of the sanitation zone from a frame of the one or more successive frames, and comparing the extracted image of the sanitation zone with a reference image of the sanitation zone to detect whether the extracted image of the sanitation zone indicates occupation of the sanitation zone; or
b) extracting, for two or more successive frames of the real time video, an extracted image of the sanitation zone from a respective frame of the two or more successive frames, and comparing a first image of the sanitation zone as extracted from a first respective frame of the two or more successive frames with a second image of the sanitation zone as extracted from a second respective frame of the two or more successive frames to detect frame-to-frame changes that indicate occupation of the sanitation zone.

### Claim 7 (depends on 1)

The hand hygiene compliance enforcement device of claim 1,
wherein the at least one annunciator comprises a noncompliance light, and
wherein the electronic processor is configured to execute the sanitation detection instructions to cause the electronic processor to perform at least one of:
a) controlling the noncompliance light to illuminate starting at a beginning of the occupancy time interval and ending at the earlier of (i) a stop time respective to an end of the occupancy time interval, or (ii) a time determined when a hand sanitation event occurs; or
b) controlling the noncompliance light to illuminate starting at an end of the occupancy time interval upon a determination that no hand sanitation event occurred during the occupancy time interval.

### Claim 8 (depends on 7)

The hand hygiene compliance enforcement device of claim 7,
wherein the at least one annunciator further comprises a compliance light, and
wherein the electronic processor is configured to execute the sanitation detection instructions to cause the electronic processor to perform at least one of:
a) controlling the compliance light to illuminate starting at a time in the occupancy time interval upon a determination that a hand sanitation event occurs;
b) controlling the compliance light to illuminate starting at an end of the occupancy time interval upon a determination that a hand sanitation event occurred during the occupancy time interval.

### Claim 9 (depends on 7)

The hand hygiene compliance enforcement device of claim 7, wherein the noncompliance light is illuminated by a flashing illumination.

### Claim 10 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the at least one annunciator comprises at least one of:
a light;
an audio speaker;
a display; or
a touchscreen display.

### Claim 11 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the instructions are configured to lack or filter identification of a person in the sanitation zone.

### Claim 12 (depends on 11)

The hand hygiene compliance enforcement device of claim 11, wherein the instructions comprise reporting instructions readable and executable by the electronic processor, the reporting instructions that when executed by the electronic processor, cause the electronic processor to:
accumulate data statistics for a plurality of successive occupancy time intervals on corresponding determinations of whether a hand sanitation event occurs and generate a compliance report on the data statistics.

### Claim 13 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the instructions comprise reporting instructions readable and executable by the electronic processor, the reporting instructions that when executed by the electronic processor, cause the electronic processor to:
identify a person in the sanitation zone during the occupancy time interval using (i) a radio frequency identification (RFID) reader that detects RFID badges, or (ii) facial recognition applied to the real time video;
accumulate data statistics for a plurality of successive occupancy time intervals on corresponding determinations of whether a hand sanitation event occurs; and
generate a compliance report based on the data statistics including identification of one or more persons identified in the sanitation zone during occupancy time intervals for which the corresponding determination of whether a hand sanitation event occurs is that no hand sanitation event occurred.

### Claim 14 (depends on 1)

The hand hygiene compliance enforcement device of claim 1, wherein the at least one video camera consists of a single video camera and the wash station zone is contained in the sanitation zone imaged by the real time video.

### Claim 15 (depends on 1)

The hand hygiene compliance enforcement device of claim 1 further comprising:
a hand sanitation station including at least a sanitation fluid dispenser,
wherein the at least one video camera is positioned such that at least a portion of the hand sanitation station coincides with the wash station zone in the real time video acquired by the at least one video camera.

### Claim 16

A non-transitory storage medium storing instructions for performing a hand hygiene compliance enforcement, instructions being readable and executable by an electronic processor, wherein the instructions, when executed by the electronic processor, cause the electronic processor to:
receive real time video acquired by at least one video camera in real time;
analyze the real time video to determine an occupancy time interval over which a sanitation zone imaged by the real time video is occupied;
determine whether a hand sanitation event occurs based on images of a wash station zone as extracted from the real time video acquired during the occupancy time interval; and
control at least one annunciator to annunciate a hand hygiene compliance status based on the determination of whether a hand sanitation event occurs.

### Claim 17 (depends on 16)

The non-transitory storage medium of claim 16, wherein determination of whether the hand sanitation event occurs comprises:
applying a convolutional neural network (CNN) to the images of the wash station zone as extracted from successive frames of the real time video acquired during the occupancy time interval to generate per-frame confidence values; and
determining whether the hand sanitation event occurs based on the per-frame confidence values,
wherein the CNN is pre-trained to generate confidence values indicating confidence that input images depict a hand sanitation event using training images of the wash station zone or a standard wash station zone that depict a hand sanitation event and training images of the wash station zone or the standard wash station zone that do not depict a hand sanitation event.

### Claim 18 (depends on 17)

The non-transitory storage medium of claim 17 wherein application of the CNN comprises:
converting the image of the wash station zone as extracted from a frame of the successive frames to a tensor of channel arrays wherein each channel array represents a color channel of the image of the wash station zone; and
applying the CNN to the tensor of channel arrays.

### Claim 19 (depends on 17)

The non-transitory storage medium of claim 17, wherein the instructions, when executed by the electronic processor, further cause the electronic processor to:
accumulate data statistics for a plurality of successive occupancy time intervals on corresponding determinations of whether a hand sanitation event occurs; and
generate a compliance report based on the data statistics.

### Claim 20

A hand hygiene compliance enforcement method comprising:
acquiring, via a at least one video camera, real time video of a field of view that includes a sanitation station having a wash station zone;
determining, via an electronic processor, whether a hand sanitation event occurs based on images of the wash station zone as extracted from the real time video; and
annunciating, via an annunciator, a compliance status based on the determination of whether a hand sanitation event occurs.

### Claim 21 (depends on 20)

The hand hygiene compliance enforcement method of claim 20, wherein determination of whether a hand sanitation event occurs comprises:
applying a machine learning (ML) classifier to the images of the wash station zone as extracted from successive frames of the real time video to generate per-frame confidence values; and
determining whether a hand sanitation event occurs based on the per-frame confidence values for the successive frames,
wherein the ML classifier is pre-trained using training images of the wash station zone or a standard wash station zone that depict a hand sanitation event and training images of the wash station zone or the standard wash station zone that do not depict a hand sanitation event.

### Claim 22 (depends on 21)

The hand hygiene compliance enforcement method of claim 21, wherein application of the ML classifier comprises:
converting an image of the wash station zone as extracted from the frame of the successive frames to a tensor of channel arrays, wherein each channel array represents a color channel of the image of the wash station zone; and
applying the ML classifier to the tensor of channel arrays.

### Claim 23 (depends on 20)

The hand hygiene compliance enforcement method of claim 20 further comprising analyzing the real time video to determine an occupancy time interval over which a sanitation zone imaged by the real time video is occupied,
wherein the determination of whether a hand sanitation event occurs is based on images of the wash station zone as extracted from only the real time video acquired during the occupancy time interval.

### Claim 24 (depends on 20)

The hand hygiene compliance enforcement method of claim 20 further comprising determining, with the electronic processor, whether a gowning event occurs based on images of a gowning zone as extracted from the real time video,
wherein the compliance status is further based on the determination of whether a gowning event occurs.

### Claim 25 (depends on 24)

The hand hygiene compliance enforcement method of claim 24, wherein determination of whether the gowning event occurs comprises:
applying a machine learning (ML) classifier to the images of the gowning zone as extracted from successive frames of the real time video to generate per-frame gowning event confidence values; and
determining whether the gowning event occurs based on the per-frame gowning event confidence values for the successive frames,
wherein the ML classifier for generating the per-frame gowning event confidence values is pre-trained using training images of the gowning zone or a standard gowning zone that depict a gowning event and training images of the gowning zone or the standard gowning zone that do not depict a gowning event.

### Claim 26

A hand hygiene compliance enforcement device comprising:
at least one non-camera-based sensor;
at least one annunciator;
an electronic processor; and
a non-transitory storage medium storing instructions that are readable and executable by the electronic processor, wherein the instructions, when executed by the electronic processor, cause the electronic processor to:
process real time data output of the non-camera-based sensor in real time,
analyze, by execution of occupancy detection instructions, the data output to determine an occupancy time interval over which a sanitation zone determined by the data output is occupied,
determine, by execution of sanitation detection instructions, whether a hand sanitation event occurs based on the data output corresponding to a wash station zone, the data output acquired during the occupancy time interval, and
control the at least one annunciator to annunciate a hand hygiene compliance status based on the hand sanitation event.

### Claim 27 (depends on 26)

The hand hygiene compliance enforcement device of claim 26, wherein the electronic processor is configured to execute the sanitation detection instructions to cause the electronic processor to:
apply, for each series of data output of the non-camera-based sensor as acquired during the occupancy time interval, a machine learning (ML) classifier to sensor data corresponding to the wash station zone, to generate a confidence value indicative of likelihood that the sensor data of the wash station zone corresponds a hand sanitation event, and
determine whether a hand sanitation event occurs in the wash station zone based on the confidence values for the successive frames,
wherein the ML classifier is pre-trained to generate the confidence value using training sensor data of the wash station zone or a standard wash station zone that defines a hand sanitation event and sensor data of the wash station zone or the standard wash station zone that do not correspond to a hand sanitation event.

### Claim 28 (depends on 27)

The hand hygiene compliance enforcement device of claim 27, wherein application of the ML classifier comprises:
converting sensor data of the wash station zone to a tensor of channel arrays, wherein each channel array emulates a color channel of an image of the wash station zone; and
applying the ML classifier to the tensor of channel arrays.

### Claim 29 (depends on 1)

The hand hygiene compliance enforcement device of claim 1 further comprising an information sharing application comprising an information sharing interface, the information sharing interface configured to display sanitation or hygiene compliance data or information.

### Claim 30 (depends on 1)

The hand hygiene compliance enforcement device of claim 1 further comprising a hand hygiene marketing platform comprising a marketing interface, the marketing interface configured to display advertisements to users in the sanitation zone or a marketing zone.