# US 2020/0394384 A1 · Real-time Aerial Suspicious Analysis (ASANA) System and Method for Identification of Suspicious individuals in public areas

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394384 A1
- Document ID: US20200394384A1
- Status: Published application
- Kind: A1
- Application number: 16/895,515
- Filed: 2020-06-08
- Published: 2020-12-17
- Earliest claimed: 2019-06-14
- Inventors: AMARJOT SINGH
- Web page: https://patentgrep.com/patent/US20200394384A1

## CPC

- G06T 7/73: using feature-based methods
- G06F 18/2431: Multiple classes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/13: Satellite images
- G06V 20/17: taken from planes or by drones
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06T 2207/30244: Camera pose
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A real-time aerial suspicious analysis (ASANA) system and method for identifying individuals engaged in suspicious activities in public areas is provided in the present invention. The aerial suspicious analysis (ASANA) uses a Drone for constant capturing and recording images/videos, and/or can be activated to capture/record based on a specific schedule and/or event; a YOLO detector to detect the individuals; a SHDL network for individual pose estimation, and then classification is performed of the estimated pose to identify the suspicious/violent individuals.

## Claims

### Claim 1

An aerial suspicious analysis (ASANA) system for identifying suspicious individuals in public areas or in a controlled environment, the system comprising:
at least one drone configured for capturing/recording one or more aerial images;
at least one computing system for performing analysis on the aerial images for extracting features from the captured/recorded image;
a YOLO detector for detecting the individuals by performing analysis on the aerial images for extracting features from the captured/recorded image;
a ScatterNet Hybrid Deep Learning (SHDL) Network for pose estimation of the detected individuals, where the ScatterNet Hybrid Deep Learning (SHDL) Network identifies fourteen key-points of a human body to form a skeleton structure of the detected individuals; and
a three dimensional (3D) ResNet for classification of the estimated pose to determine whether the suspicious individuals exist in the estimated pose,
wherein the ScatterNet Hybrid Deep Learning (SHDL) Network is trained with an Aerial Violent Individual (AVI) Dataset to perform analysis of the identified key-points and the 3D ResNet is trained on the estimated skeletons for at least five suspicious or violent activities (Punching, Stabbing, Shooting, Kicking, and Strangling) and one neutral activity, in the Aerial Violent Individual (AVI) Dataset, to perform multi-class classification
wherein the Aerial Violent Individual (AVI) Dataset is composed of thousands of images and thousands of individuals engaged in one or more suspicious or violent activities.

### Claim 2 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, further provides monitoring such as but limited to criminal activities, abnormal events or incidents by the individuals.

### Claim 3 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the drone is further configured to monitor a coverage area to detect incidents occurring within and/or approximate to the coverage area and respond to these incidents.

### Claim 4 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the drone is configured to perform constant capturing/recording, and/or can be activated to capture/record based on a specific schedule and/or event

### Claim 5 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein further configured with a processing device for onboard processing or processing on a cloud server to perform computations in real-time for identifying the suspicious individuals.

### Claim 6 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the system is preconfigured with the Aerial Violent Individual (AVI) Dataset or the Aerial Violent Individual (AVI) Dataset is used to train a statistical or a machine learning model for the system.

### Claim 7 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the Aerial Violent Individual (AVI) Dataset includes images with various individuals recorded at different variations of scale, position, illumination, blurriness, etc.

### Claim 8 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the Aerial Violent Individual (AVI) Dataset consist of thousands of individuals engaged in one or more suspicious or violent activities such as but not limited to Punching, Stabbing, Shooting, Kicking, Strangling Pushing, Shoving, Grabbing, Slapping, Physically assaulting, Hitting etc.

### Claim 9 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the ScatterNet Hybrid Deep Learning (SHDL) uses orientations of limbs to estimate the pose of the individuals.

### Claim 10 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the 3D ResNet uses the estimated poses to identify the suspicious individuals.

### Claim 11 (depends on 1, 10)

The aerial suspicious analysis (ASANA) system of claims 1 and 10, wherein the 3D ResNet classifies the individuals as either neutral or assigns a most likely suspicious or violent activity label using the estimated poses.

### Claim 12 (depends on 1)

The aerial suspicious analysis (ASANA) system of claim 1, wherein the fourteen key-points are annotated on the human body as Facial Region (P1—Head region, P2—Neck), Arms Region (P3—Right shoulder, P4—Right Elbow, P5—Right Wrist, P6—Left Shoulder, P7—Left Elbow, P8—Left Wrist) and Legs Region (P9—Right Hip, P10—Right Knee, P11—Right Ankle, P12—Left Hip, P13—Left Knee, P14—Left Ankle).

### Claim 13

A method for identifying suspicious individuals in public areas or in a controlled environment, the method comprising:
capturing/recording one or more aerial images using one or more drones;
performing analysis on the aerial images for extracting features from the captured/recorded image;
detecting individuals using a YOLO detector;
pose estimation of the individuals using a ScatterNet Hybrid Deep Learning (SHDL) Network;
identifying fourteen key-points of a human body to form a skeleton structure of the detected individuals; and
classifying of the estimated pose using a three dimensional (3D ResNet for determining whether the suspicious individuals exist in the estimated pose,
wherein the ScatterNet Hybrid Deep Learning (SHDL) Network is trained with an Aerial Violent Individual (AVI) Dataset to perform analysis of the identified key-points, and the 3D ResNet is trained on the estimated skeletons for at least five suspicious or violent activities (Punching, Stabbing, Shooting, Kicking, and Strangling) and one neutral activity, in the Aerial Violent Individual (AVI) Dataset, to perform multi-class classification
wherein the Aerial Violent Individual (AVI) Dataset is composed of thousands of images and thousands of individuals engaged in one or more suspicious or violent activities.

### Claim 14 (depends on 13)

The method of claim 13, further includes monitoring such as but limited to criminal activities, abnormal events or incidents by the individuals.

### Claim 15 (depends on 13)

The method of claim 13, further includes monitoring a coverage area to detect incidents occurring within and/or approximate to the coverage area and responding to these incidents.

### Claim 16 (depends on 13)

The method of claim 13, wherein a processing device for onboard processing or processing on a cloud server for performing computations in real-time for identifying the suspicious individuals.

### Claim 17 (depends on 13)

The method of claim 13, wherein includes identifying the suspicious individuals from a Aerial Violent Individual (AVI) Dataset, where the Aerial Violent Individual (AVI) Dataset consist of thousands of individuals engaged in one or more suspicious or violent activities such as but not limited to Punching, Stabbing, Shooting, Kicking, Strangling Pushing, Shoving, Grabbing, Slapping, Physically assaulting, Hitting etc.

### Claim 18 (depends on 13)

The method of claim 13, wherein the ScatterNet Hybrid Deep Learning (SHDL) uses orientations of limbs to estimate the pose of the individuals.

### Claim 19 (depends on 13)

The method of claim 13, wherein the 3D ResNet uses the estimated poses to identify the suspicious individuals.

### Claim 20 (depends on 13)

The method of claim 13, wherein the fourteen key-points are annotated on the human body as Facial Region (P1—Head region, P2—Neck), Arms Region (P3—Right shoulder, P4—Right Elbow, P5—Right Wrist, P6—Left Shoulder, P7—Left Elbow, P8—Left Wrist) and Legs Region (P9—Right Hip, P10—Right Knee, P11—Right Ankle, P12—Left Hip, P13—Left Knee, P14—Left Ankle).
