# US 12,183,173 B2 · Systems and methods for providing personalized and contextualized environment security information in a remediation conversation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,173 B2
- Document ID: US12183173B2
- Status: Granted
- Kind: B2
- Application number: 17/851,987
- Filed: 2022-06-28
- Published: 2023-01-12
- Granted: 2024-12-31
- Earliest claimed: 2021-07-06
- Assignees: Johnson Controls Tyco IP Holdings LLP
- Applicants: Johnson Controls Tyco IP Holdings LLP
- Inventors: Jason M. Ouellette; Gopal Paripally; Kapang Lam
- Web page: https://patentgrep.com/patent/US12183173B2

## CPC

- G08B 13/196: using television cameras
- G08B 13/19682: {Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G07C 9/28: the pass enabling tracking or indicating presence
- H04L 63/08: {for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04L 41/16: using machine learning or artificial intelligence
- H04L 41/22: comprising specially adapted graphical user interfaces [GUI]
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed herein are apparatuses and methods for providing personalized and contextualized environment security information. An implementation may comprise collecting a plurality of sensor data from a plurality of sensors located in an environment, detecting a security event in the environment based on at least a portion of the plurality of sensor data, and generating contextual information for the security event based on at least a larger portion of the plurality of sensor data. The implementation may further comprise detecting a first user accessing security information for the environment on an output device, retrieving a user profile of the first user, and generating a remediation conversation based on the user profile, the security event, and the contextual information. The implementation may further comprise outputting at least a first portion of the remediation conversation on the output device.

## Claims

### Claim 1

An apparatus for providing personalized and contextualized environment security information, comprising:
a memory;
a processor coupled with the memory and configured to:
collect a plurality of sensor data from a plurality of sensors located in an environment;
detect a security event in the environment based on at least a portion of the plurality of sensor data;
generate contextual information for the security event based on at least a larger portion of the plurality of sensor data;
detect a first user accessing security information for the environment on an output device;
retrieve a user profile of the first user, wherein the user profile is indicative of access rights and user interface preferences;
generate a remediation conversation based on the user profile, the security event, and the contextual information, wherein the remediation conversation comprises dialogue for providing security information related to the first user and guiding the first user to resolve the security event, wherein another remediation conversation about the security event that is presented to a second user does not include the security information related to the first user; and
output at least a first portion of the remediation conversation on the output device.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to:
receive, from the first user, a user response to the remediation conversation, wherein the user response requests additional information pertaining to the security event;
identify a subset of the sensors in the environment that generated data used to formulate the security event;
collect additional sensor data from the subset or the sensors; and
output a second portion of the remediation conversation.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the user response further requests performing object tracking on an object in the environment, wherein the processor is further configured to output another portion of the remediation conversation by:
identifying the object using the additional sensor data;
collecting the additional sensor data until the object is no longer detected; and
generating the tracked object for display.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein the user response is one of: a verbal response, a gesture, a physical input, and an expression.

### Claim 5 (depends on 2)

The apparatus of claim 2, wherein the processor is further configured to:
receive another user response from the first user; and
output a third portion of the remediation conversation, wherein portions of the remediation conversation are outputted until the security event is resolved or the first user ceases providing user responses.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the remediation conversation for the first user is different from the another remediation conversation of the second user having at least one of a different access rights and a different user interface preferences.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein at least the portion of the plurality of sensor data comprises sensor data from one sensor of the plurality of sensors and wherein at least the larger portion of the plurality of sensor data comprises sensor data from one or more different sensors.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the contextual information comprises a cause of the security event, and wherein the processor is further configured to generate the contextual information for the security event based on at least the larger portion of the plurality of sensor data by:
determining the cause of the security event by comparing at least the larger portion of the plurality of sensor data with activity templates comprising sensor data during historic activities and associated activity identifiers.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the remediation conversation comprises options for resolving the security event, wherein different options are provided by the remediation conversation for different contextual information.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the user interface preferences comprises a preferred medium of communication, wherein a medium includes audio, video, and/or physical feedback,
wherein audio preferences include at least one of:
a preferred language,
a preferred voice output,
a preferred speech speed;
wherein video preferences include at least one of:
an appearance of a user interface where the remediation conversation is generated,
a video quality,
wherein physical feedback preferences include at least one of:
a touchscreen sensitivity of the output device,
a vibration strength of the output device,
a haptic feedback sensitivity of the output device.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the security event is indicative of a deviation from a known trend in the environment, wherein the processor is further configured to:
execute a machine learning algorithm to classify whether input sensor data comprises the deviation by:
generating a feature vector representing the known trend in historic sensor data;
comparing the feature vector against an input feature vector of the input sensor data; and
classifying the input sensor data as comprising the deviation in response to determining, based on the comparison, that a difference between the feature vector and the input feature vector is greater than a threshold difference.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the security event is indicative of an inconsistency between at least two sensors, wherein the processor is further configured to:
receive a first sensor output from a first sensor in the environment, and a second sensor output from a second sensor in the environment; and
determine, based on historical sensor data, that the first sensor should not output the first sensor output when the second sensor outputs the second sensor output.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein the security event represents a summary of monitored activity over a period of time from at least one sensor of the plurality of sensors.

### Claim 14

A method for providing personalized and contextualized environment security information, comprising:
collecting a plurality of sensor data from a plurality of sensors located in an environment;
detecting a security event in the environment based on at least a portion of the plurality of sensor data;
generating contextual information for the security event based on at least a larger portion of the plurality of sensor data;
detecting a first user accessing security information for the environment on an output device;
retrieving a user profile of the first user, wherein the user profile is indicative of access rights and user interface preferences;
generating a remediation conversation based on the user profile, the security event, and the contextual information, wherein the remediation conversation comprises dialogue for providing security information related to the first user and guiding the first user to resolve the security event, wherein another remediation conversation about the security event that is presented to a second user does not include the security information related to the first user; and
outputting at least a first portion of the remediation conversation on the output device.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
receiving, from the first user, a user response to the remediation conversation, wherein the user response requests additional information pertaining to the security event;
identifying a subset of the sensors in the environment that generated data used to formulate the security event;
collecting additional sensor data from the subset of the sensors; and
outputting a second portion of the remediation conversation.

### Claim 16 (depends on 15)

The method of claim 15, wherein the user response further requests performing object tracking on an object in the environment, wherein outputting another portion of the remediation conversation further comprises:
identifying the object using the additional sensor data;
collecting the additional sensor data until the object is no longer detected; and
generating the tracked object for display.

### Claim 17 (depends on 15)

The method of claim 15, wherein the user response is one of: a verbal response, a gesture, a physical input, and an expression.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
receiving another user response from the first user; and
outputting a third portion of the remediation conversation, wherein portions of the remediation conversation are outputted until the security event is resolved or the first user ceases providing user responses.

### Claim 19 (depends on 14)

The method of claim 14, wherein the remediation conversation for the first user is different from the another remediation conversation of the second user having at least one of a different access rights and a different user interface preferences.

### Claim 20

A non-transitory computer-readable medium storing instructions, executable by a processor, for performing a method for providing personalized and contextualized environment security information, comprising:
collecting a plurality of sensor data from a plurality of sensors located in an environment;
detecting a security event in the environment based on at least a portion of the plurality of sensor data;
generating contextual information for the security event based on at least a larger portion of the plurality of sensor data;
detecting a first user accessing security information for the environment on an output device;
retrieving a user profile of the first user, wherein the user profile is indicative of access rights and user interface preferences;
generating a remediation conversation based on the user profile, the security event, and the contextual information, wherein the remediation conversation comprises dialogue for providing security information related to the first user and guiding the first user to resolve the security event, wherein another remediation conversation about the security event that is presented to a second user does not include the security information related to the first user; and
outputting at least a first portion of the remediation conversation on the output device.