# US 2020/0402419 A1 · System and method for automatically recognizing activities and gaze patterns in human patient simulations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402419 A1
- Document ID: US20200402419A1
- Status: Published application
- Kind: A1
- Application number: 16/907,496
- Filed: 2020-06-22
- Published: 2020-12-24
- Earliest claimed: 2019-06-21
- Assignees: Cleveland State University
- Applicants: Cleveland State University
- Inventors: Wenbing Zhao; William A. Matcham
- Web page: https://patentgrep.com/patent/US20200402419A1

## CPC

- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G09B 19/003: {Repetitive work cycles; Sequence of movements}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 20/20: in augmented reality scenes
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G09B 9/00: Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131})
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G09B 23/30: Anatomical models {(G09B23/281 - G09B23/288 take precedence)}
- G06T 2207/30201: Face
- G06T 2207/30204: Marker
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method automatically recognizes complex activities and gaze patterns. One particular application of the system and method is the recognition of the activities and gaze patterns of a nursing student who practices nursing skills during human patient simulations (HPS). Accordingly, the system and method provide a novel event-drive context model for complex human activity and gaze recognition. The system and method include the identification of all necessary context and the development of an ontology for HPS to recognize complex activities in HPS, the encoding of correctness rules for skills in HPS, automated identification of errors made by the student during a simulation session, and the development of personalized feedback based on the data collected during a session and the predefined correctness rules. The system and method also incorporates a marker-based mechanism that facilitates privacy-aware tracking wherein only a consented user will be monitored.

## Claims

### Claim 1

A system for training a trainee, the system comprising:
mixed-reality hardware, wearable by the trainee and having a display viewable by the trainee;
a camera coupled to a tracking logic;
an object of interest, the object of interest having a marker associated therewith; and
a training logic;
wherein the mixed reality hardware is configured to:
track a position of the object of interest based on the marker,
on the display, overlay an image onto the object of interest based on the marker,
detect whether the trainee is gazing at the object of interest, and
detect human speech;
wherein the tracking logic is configured to track skeletal movements of the trainee; and
wherein the training logic is configured to provide feedback to the trainee regarding training progress based on:
tracking of the object of interest,
detecting whether the trainee is gazing at the object of interest,
detecting human speech, and
tracking the skeletal movements of the trainee.
