# US 12,602,941 B2 · System and method for identifying atypical events and generating an alert using deep learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,941 B2
- Document ID: US12602941B2
- Status: Granted
- Kind: B2
- Application number: 17/871,901
- Filed: 2022-07-23
- Published: 2024-01-25
- Granted: 2026-04-14
- Earliest claimed: 2022-07-23
- Assignees: International Institute of Information Technology, Hyderabad
- Applicants: International Institute of Information Technology, Hyderabad
- Inventors: Ravi Kiran Sarvadevabhatla
- Web page: https://patentgrep.com/patent/US12602941B2

## CPC

- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 11/23: using straight lines or curves
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 30/1801: {Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections (extracting features by contour coding G06V30/182)}
- G06V 30/347: {Sampling; Contour coding; Stroke extraction}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G08B 5/22: using electric transmission; using electromagnetic transmission
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

The embodiments herein relate to a system and a method identifying and generating an alert, using a deep learning model, for atypical events during an activity on an electronic whiteboard. The system includes a user device associated with a user, and an atypical event detecting server. The atypical event detecting server (i) receives one or more canvases at concurrent sessions, (ii) annotates the one or more canvases, (iii) detects historical atypical events from the annotated one or more canvases, (iv) trains the deep learning model by correlating the historical atypical events, historical bounding boxes with historical canvases in historical sessions, (v) processes the one or more canvases by determining a bounding box around each atypical event, and (vi) generates an alert for identified atypical events using an event record table.

## Claims

### Claim 1

A processor-implemented method for identifying and generating an alert, using a deep learning model, for a plurality of atypical events during an activity on an electronic whiteboard using a user device associated with a user, the method comprising:
receiving a plurality of canvases at concurrent sessions from a user device associated with a user, wherein each canvas comprises strokes, wherein each stroke is a timestamped scalable vector graphics element;
annotating the plurality of canvases by (i) displaying the strokes from each canvas of a session, (ii) assigning labels for the strokes by selecting at least one category related to any of the plurality of atypical events present, (iii) enabling an instance-level annotation for each category that comprises the strokes to select a contiguous group of sketches, and (iv) high-lighting the strokes belonging to the contiguous group of sketches to obtain annotated plurality of canvases;
detecting historical atypical events from the annotated plurality of canvases by (i) detecting, using an image-based object detection, the high-lighted strokes of the contiguous group of sketches to represent as spatially localized objects (ii) extracting a plurality of features from the spatially localized objects, and (iii) generating feature maps from the extracted features;
training the deep learning model by correlating the historical atypical events, historical bounding boxes with historical canvases in historical sessions to obtain a trained deep learning model;
processing, using the deep learning model, the plurality of canvases by determining a bounding box around each atypical event, thereby identifying the plurality of atypical events during the activity on the electronic whiteboard using the bounding box around each atypical event; and
generating, using an event record table, the alert for identified plurality of atypical events by (i) recording atypical event information related to on-going sessions, (ii) analyzing, using a rule base method, the atypical event information to generate an appropriate alert by relaying to an appropriate session, thereby displaying the appropriate alert on a user interface of the user device associated with the user.

### Claim 2 (depends on 1)

The processor-implemented method of claim 1, wherein the method comprises optimizing the trained deep learning model by calculating (i) a classification loss using confidence scores between ground truth offsets and predicted offsets, and (ii) a bounding-box localization loss using distance intersection over union (IoU) loss.

### Claim 3 (depends on 1)

The processor-implemented method of claim 1, wherein the method further comprises,
grouping the strokes of the historical atypical events into subsequences that are separated by erase stroke subsequences;
converting curves of the strokes to a point sequence representation;
down sampling, using ramer-douglas-peucker method, the point sequence representation adaptively to obtain points;
rendering the points on a canvas for data annotation and representation; and
labelling the annotated and represented points to automatically generate the ground truth offsets to train the deep learning model.

### Claim 4 (depends on 1)

The processor-implemented method of claim 1, wherein the method further comprises, augmenting the historical atypical events in the historical sessions to train the deep learning model by,
isolating subsequences of the strokes of the historical atypical events;
sampling the subsequences of the strokes of a current session and adding the subsequences of the strokes with the next sessions that do not contain the historical atypical events to obtain sampled subsequences; and
ensuring the sampled subsequences are spatially disjoint from the strokes of the sessions by localizing the sampled subsequences.

### Claim 5 (depends on 4)

The processor-implemented method of claim 4, wherein the plurality of atypical events comprises at least one of text, numericals, circles, or iconics, wherein the at least one category related to any of the plurality of atypical events present in the menu bar of an annotation interface comprises running hand letters, circles, arrows, question marks, sketch, individual letter, numbers.

### Claim 6 (depends on 4)

The processor-implemented method of claim 4, wherein the method further comprises,
grouping the strokes into consecutive subsequences of erase strokes; and
generating a new session by taking stroke subsequence from a first stroke of the concurrent sessions to a first stroke of one of erase subsequences.

### Claim 7

One or more non-transitory computer-readable storage medium storing the one or more sequence of instructions, which when executed by the one or more processors, causes to perform a method for identifying and generating an alert for a plurality of atypical events on an electronic whiteboard using a deep learning model, comprising:
receiving a plurality of canvases at concurrent sessions from a user device associated with a user, wherein each canvas comprises strokes, wherein each stroke is a timestamped scalable vector graphics element;
annotating the plurality of canvases by (i) displaying the strokes from each canvas of a session, (ii) assigning labels for the strokes by selecting at least one category related to any of the plurality of atypical events, (iii) enabling an instance-level annotation for each category that comprises the strokes to select a contiguous group of sketches, and (iv) high-lighting the strokes belonging to the contiguous group of sketches to obtain annotated plurality of canvases;
detecting historical atypical events from the annotated plurality of canvases by (i) detecting, using an image-based object detection, the high-lighted strokes of the contiguous group of sketches to represent as spatially localized objects (ii) extracting a plurality of features from the spatially localized objects, and (iii) generating feature maps from the extracted features;
training the deep learning model by correlating the historical atypical events, historical bounding boxes with historical canvases in historical sessions to obtain a trained deep learning model;
processing, using the deep learning model, the plurality of canvases by determining a bounding box around each atypical event, thereby identifying the plurality of atypical events during the activity on the electronic whiteboard using the bounding box around each atypical event; and
generating, using an event record table, an alert for identified plurality of atypical events by (i) recording atypical event information related to on-going sessions, (ii) analyzing, using a rule base method, the atypical event information to generate an appropriate alert by relaying to an appropriate session, thereby displaying the appropriate alert on an user interface of the user device associated with the user.

### Claim 8

A system for identifying and generating an alert for a plurality of atypical events on an electronic whiteboard using a deep learning model, the system comprising:
a device processor; and
a non-transitory computer-readable storage medium storing one or more sequences of instructions, which when executed by the device processor, causes
receive a plurality of canvases at concurrent sessions from a user device associated with a user, wherein each canvas comprises strokes, wherein each stroke is a timestamped scalable vector graphics element;
annotate the plurality of canvases by (i) displaying the strokes from each canvas of a session, (ii) assigning labels for the strokes by selecting at least one category related to any of the plurality of atypical events, (iii) enabling an instance-level annotation for each category that comprises the strokes to select a contiguous group of sketches, and (iv) high-lighting the strokes belonging to the contiguous group of sketches to obtain annotated plurality of canvases;
detect historical atypical events from the annotated plurality of canvases by (i) detecting, using an image-based object detection, the high-lighted strokes of the contiguous group of sketches to represent as spatially localized objects (ii) extracting a plurality of features from the spatially localized objects, and (iii) generating feature maps from the extracted features;
train the deep learning model by correlating the historical atypical events, historical bounding boxes with historical canvases in historical sessions to obtain a trained deep learning model;
process, using the deep learning model, the plurality of canvases by determining a bounding box around each atypical event, thereby identifying the plurality of atypical events during the activity on the electronic whiteboard using the bounding box around each atypical event; and
generate, using an event record table, an alert for identified plurality of atypical events by (i) recording atypical event information related to on-going sessions, (ii) analyzing, using a rule base method, the atypical event information to generate an appropriate alert by relaying to an appropriate session, thereby displaying the appropriate alert on an user interface of the user device associated with the user.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processor is configured to optimize the trained deep learning model by calculating (i) a classification loss using confidence scores between ground truth offsets and predicted offsets, and (ii) a bounding-box localization loss using distance intersection over union (IoU) loss.

### Claim 10 (depends on 8)

The system of claim 8, wherein the processor is configured to;
group the strokes of the historical atypical events into subsequences that are separated by erase stroke subsequences;
convert curves of the strokes to a point sequence representation;
down sample, using ramer-douglas-peucker method, the point sequence representation adaptively to obtain points;
render the points on a canvas for data annotation and representation;
label the annotated and represented points to automatically generate ground-truth data to train the deep learning model.

### Claim 11 (depends on 8)

The system of claim 8, wherein the processor is configured to augment the historical atypical events in the historical sessions to train the deep learning model by,
isolating subsequences of the strokes of the historical atypical events;
sampling the subsequences of the strokes of a current session and adding the subsequences of the strokes with the next sessions that do not contain the historical atypical events to obtain sampled subsequences; and
ensuring the sampled subsequences are spatially disjoint from the strokes of the sessions by localizing the sampled subsequences.

### Claim 12 (depends on 11)

The system of claim 11, wherein the processor is configured to segment the received image into the plurality of classes by,
extracting, using an encoder, the received image into at least one feature map to capture a plurality of features, wherein the plurality of features are captured using filters; and
recovering, using a decoder, a resolution of the at least one feature map to obtain the plurality of classes.

### Claim 13 (depends on 8)

The system of claim 8, wherein the plurality of atypical events comprises at least one of text, numericals, circles, or iconics, wherein the at least one category related to any of the plurality of atypical events present in the menu bar of an annotation interface comprises running hand letters, circles, arrows, question marks, sketch, individual letter, numbers.

### Claim 14 (depends on 8)

The system of claim 8, wherein the processor is configured to:
grouping the strokes into consecutive subsequences of erase strokes; and
generating a new session by taking stroke subsequence from a first stroke of the concurrent sessions to a first stroke of one of erase subsequences.