# US 11,307,667 B2 · Systems and methods for facilitating accessible virtual education

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,667 B2
- Document ID: US11307667B2
- Status: Granted
- Kind: B2
- Application number: 16/892,248
- Filed: 2020-06-03
- Published: 2020-12-03
- Granted: 2022-04-19
- Earliest claimed: 2019-06-03
- Assignees: Arizona Board of Regents on behalf of Arizona State University
- Applicants: Ayan Banerjee; Sandeep Gupta; Prajwal Paudyal
- Inventors: Ayan Banerjee; Sandeep Gupta; Prajwal Paudyal
- Web page: https://patentgrep.com/patent/US11307667B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 18/23: Clustering techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 40/107: {Static hand or arm}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G09B 5/067: {Combinations of audio and projected visual presentation, e.g. film, slides}
- G09B 5/14: with provision for individual teacher-student communication
- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G06N 3/045: Combinations of networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Various embodiments of systems and methods for a learning environment for accessible virtual computer science education are disclosed.

## Claims

### Claim 1

A method, comprising,
generating an expert reach set for a gesture as performed by a tutor wherein the expert reach set defines correctness for transient movement dynamics and hand shape for the gesture;
receiving a video feed of a user performing the gesture;
converting the gesture as performed by the user into a spatio-temporal evolution of continuous variables by:
generating an X and Y coordinate time series, wherein the positional X and Y coordinate time series is representative of positions of a right wrist and a left wrist of the user; and
extracting a hand shape of the user; and
determining a deviation of the gesture as performed by the user from the expert reach set for the gesture; and
providing correctness feedback to the user based on the deviation of the gesture as performed by the user from the expert reach set for the gesture,
wherein the expert reach set is a set of continuous states observed from simulating a hybrid system representative of the gesture over time for a bounded set of initial conditions
wherein the hybrid system is determined by:
segmenting the gesture as captured by the video feed of the user into a plurality of discrete modes:
clustering each mode of the plurality of discrete modes in accordance with their respective triggering mechanism;
obtaining one or more kinematic equations representative of the gesture by:
processing the gesture data using a multi-variate polynomial regression model or Fisher Information with Cramer Rao bound to obtain a set of flow parameters representative of mode transitions; and
clustering the gesture data using a density based approach to the set of flow parameters for each mode of the plurality of modes; and
deriving guard conditions for each cluster of the gesture.

### Claim 2 (depends on 1)

The method of claim 1, wherein the step of generating an expert reach set for a gesture further comprises:
deriving a plurality of initial conditions for the gesture as performed by the tutor by performing a statistical analysis of the tutor's speed, initial positions, and kinematic model parameters; and
performing a reachability analysis using the hybrid system representative of the gesture as performed by the tutor.

### Claim 3 (depends on 1)

The method of claim 1, wherein the X and Y coordinate time series is representative of position of the user's wrists over time and wherein the X and Y coordinate time series is extracted using a plurality of location buckets defined around a face and a chest of the user, as captured by the video feed of the user.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
training a neural network having a plurality of layers using gesture hand shapes;
processing the hand shape of the tutor and the hand shape of the user using the neural network; and
comparing respective outputs of a penultimate layer of the plurality of layers using a Euclidean distance metric, wherein each output is representative of the hand shape of the tutor and the hand shape of the user.