# US 11,443,515 B2 · Systems and methods for machine learning enhanced intelligent building access endpoint security monitoring and management

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,515 B2
- Document ID: US11443515B2
- Status: Granted
- Kind: B2
- Application number: 16/696,682
- Filed: 2019-11-26
- Published: 2020-06-25
- Granted: 2022-09-13
- Earliest claimed: 2018-12-21
- Assignees: Ambient AI, Inc.
- Applicants: Ambient AI, Inc.
- Inventors: Shikhar Shrestha; Vikesh Khanna; James Douglas Connor
- Web page: https://patentgrep.com/patent/US11443515B2

## CPC

- G06F 21/554: {involving event detection and direct action}
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/2431: Multiple classes
- G06F 21/31: User authentication
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 5/04: Inference or reasoning models
- G06N 20/20: Ensemble learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 30/19173: {Classification techniques}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G08B 13/19613: {Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion}
- G08B 13/19671: {Addition of non-video data, i.e. metadata, to video stream}
- G08B 29/186: {Fuzzy logic; neural networks}
- G08B 29/188: {Data fusion; cooperative systems, e.g. voting among different detectors}
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06N 3/045: Combinations of networks
- G06Q 50/265: {Personal security, identity or safety}
- G06V 20/44: {Event detection}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods for correlating access-system primitives generated by an access control system and semantic primitives generated by a sensor data comprehension system.

## Claims

### Claim 1

A method comprising:
generating a first access-system primitive from credential data generated by an access credential input device controlling access to a first access point;
generating a second access-system primitive from an access point state generated by an access point state sensor monitoring a state of the first access point;
generating at least one from image data generated by a video camera directed toward the first access point;
identifying at least one contextual event by correlating the first access-system primitive, the second access-system primitive, and at least one semantic primitive;
classifying each contextual event to generate at least one contextual event classification;
detecting at least one security threat event based on the at least one contextual event classification, and performing at least one security action in response to detection of each security threat event; and
wherein performing at least one security action comprises at least one of providing an alert and controlling the electronic access controller.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying at least one contextual event by correlating at least one access-system primitive with at least one semantic primitive comprises: identifying at least one access-system primitive and at least one semantic primitive associated with the first access point controlled by the electronic access controller.

### Claim 3 (depends on 1)

The method of claim 1, wherein classifying each contextual event comprises classifying each contextual event by using a contextual security event model that identifies contextual events classified as contextual security events.

### Claim 4 (depends on 3)

The method of claim 3, wherein contextual security events comprise at least one of: “compromised credential”, “terminated employee”, “unusual access pattern by validly credentialed employee”, “tailgating”, and “piggybacking”.

### Claim 5 (depends on 3)

The method of claim 3,
wherein contextual security events comprise security events not recognized by the electronic access controller,
the method further comprising: automatically generating the contextual security event model by performing autogenous correlation using at least one access-system primitive and at least one semantic primitive.

### Claim 6 (depends on 5)

The method of claim 5, further comprising: using the contextual security event model to detect compromising of a valid user credential when a correlation engine identifies access attempts to access points with the valid user credential at multiple distinct access points.

### Claim 7 (depends on 5)

The method of claim 5, further comprising: using the contextual security event model to detect site access by a terminated employee.

### Claim 8 (depends on 5)

The method of claim 5, further comprising: using the contextual security event model to detect a security threat based on an unusual access pattern to one or more access points by a validly credentialed employee.

### Claim 9 (depends on 5)

The method of claim 5, further comprising: using the contextual security event model to detect tailgating by one or more users through an access point after a valid access credential is used to enable the access point for ingress by a credentialed user.

### Claim 10 (depends on 1)

The method of claim 1,
wherein generating a plurality of semantic primitives comprises: using a high-level feature detection model to extract high-level feature values from sensor data generated by the at least one sensor data source, and processing the high-level feature values by using each sub-model of a machine learning ensemble to generate the plurality of semantic primitives,
the method further comprising: training the high-level feature extraction model based on output of each sub-model of the machine learning ensemble.

### Claim 11 (depends on 1)

The method of claim 1, wherein identifying at least one contextual event by correlating at least one access-system primitive with at least one semantic primitive comprises:
generating a distinct access-system timeline that includes access-system primitives;
generating a distinct comprehension system timeline that includes semantic primitives; and
generating a unified event timeline that includes access-system primitives included in the access-system timeline and semantic primitives included in the comprehension system timeline.

### Claim 12 (depends on 11)

The method of claim 11,
further comprising: displaying a graphical representation of the unified event timeline,
wherein classifying each contextual event comprises: generating a classification label based on one or more primitives included in the unified event timeline.

### Claim 13 (depends on 1)

The method of claim 1, further comprising detecting at least one access-system malfunction event from among the classified contextual events, and performing at least one access-system malfunction action in response to detection of each access-system malfunction event.

### Claim 14 (depends on 1)

The method of claim 1, wherein the credential data, the access point state, and the image data are contemporaneously generated.

### Claim 15 (depends on 1)

The method of claim 1, wherein the contextual event classification is determined by a machine learning classifier.

### Claim 16 (depends on 13)

The method of claim 13, wherein image data provided by the video camera arranged to capture image data of the scene that includes the first access point is used to detect at least one access-system malfunction event for the first access point.

### Claim 17 (depends on 13)

The method of claim 13, wherein detecting at least one access-system malfunction event from among the classified contextual events comprises: detecting a hardware malfunction event for at least one of the electronic access controller, an access point state sensor, a door access switch, an electronic lock, and an electronic door.

### Claim 18 (depends on 17)

The method of claim 17, wherein image data provided by the video camera arranged to capture image data of the scene that includes the first access point is used to detect a hardware malfunction event for a door access switch for the first access point.

### Claim 19 (depends on 17)

The method of claim 17, wherein image data provided by the video camera arranged to capture image data of the scene that includes the first access point is used to detect a hardware malfunction event for an electronic lock for the first access point.

### Claim 20 (depends on 17)

The method of claim 17, wherein image data provided by the video camera arranged to capture image data of the scene that includes the first access point is used to detect a hardware malfunction event for an electronic door for the first access point.

### Claim 21 (depends on 17)

The method of claim 17, wherein image data provided by the video camera arranged to capture image data of the scene that includes the first access point is used to detect a hardware malfunction event for an access point state sensor for the first access point.

### Claim 22 (depends on 17)

The method of claim 17, wherein image data provided by the video camera arranged to capture image data of the scene that includes the first access point is used to detect a hardware malfunction event for an access credential input device for the first access point.