# US 12,764,130 B2 · Monitoring hosts for anomalous processes using machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,764,130 B2
- Document ID: US12764130B2
- Status: Granted
- Kind: B2
- Application number: 18/795,280
- Filed: 2024-08-06
- Published: 2024-11-28
- Granted: 2026-10-13
- Earliest claimed: 2020-09-17
- Assignees: Rapid7, Inc.
- Applicants: Rapid7, Inc.
- Inventors: Jocelyn Beauchesne; John Lim Oh; Vasudha Shivamoggi; Roy Donald Hodgman
- Web page: https://patentgrep.com/patent/US12764130B2

## CPC

- H04L 63/1416: {Event detection, e.g. attack signature detection}
- G06N 20/00: Machine learning
- G16B 40/30: Unsupervised data analysis
- H04L 63/0823: {using certificates (cryptographic mechanisms or cryptographic arrangements for entity authentication involving certificates H04L9/3263)}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY

## Abstract

An anomaly detection system is disclosed capable of reporting anomalous processes or hosts in a computer network using machine learning models trained using unsupervised training techniques. In embodiments, the system assigns observed processes to a set of process categories based on the file system path of the program executed by the process. The system extracts a feature vector for each process or host from the observation records and applies the machine learning models to the feature vectors to determine an outlier metric each process or host. The processes or hosts with the highest outlier metrics are reported as detected anomalies to be further examined by security analysts. In embodiments, the machine learnings models may be periodically retrained based on new observation records using unsupervised machine learning techniques. Accordingly, the system allows the models to learn from newly observed data without requiring the new data to be manually labeled by humans.

## Claims

### Claim 1

A system comprising:
one or more computing devices having one or more hardware processors and memory storing program instructions that when executed by the one or more hardware processors implement an anomaly detection system and cause the anomaly detection system to:
receive a plurality of observation records indicating metadata about a plurality of processes executed on one or more hosts, wherein the processes belong to a process category;
for each observation record of a process:
extract a plurality of features of the process from the observation record to generate a feature vector; and
process the feature vector using one or more outlier detection models to determine an outlier metric for the process within the process category, wherein the one or more outlier detection models are repeatedly retrained using one or more unsupervised machine learning techniques based on a first data store containing normal observation records of the process category;
rank the processes based on respective outlier metrics of the processes;
detect a set of anomalous processes among the processes, wherein the set of anomalous processes is limited to a specified number or percentile of the processes that have highest outlier metric rankings;
detect one or more observation records of the anomalous processes in the set that indicates a signal of a potential cyberattack, wherein the signal is detected based on a scan of a second data store containing anomalous observation records associated with confirmed cyberattacks;
generate, via a graphical user interface (GUI), an alert indicating the signal; and
in response to confirmation of the signal received via the GUI:
remove the one or more observation records from the first data store used to retrain the one or more outlier detection models;
archive the one or more observation records in the second data store as an example signal of the potential cyberattack; and
initiate one or more remediation processes to mitigate against the potential cyberattack.

### Claim 2 (depends on 1)

The system of claim 1, wherein the program instructions when executed cause the anomaly detection system to:
categorize observed processes on a host into a plurality of process categories including the process category, wherein the observed processes are categorized based on one or more of:
(a) an executable used to launch the observed process,
(b) a file system path of the executable,
(c) a version of the executable,
(d) a MD5 value of the executable,
(e) a user who launched the process, and
(f) a time when the process was launched.

### Claim 3 (depends on 2)

The system of claim 2, wherein the process categories are related to one another in a hierarchical process tree.

### Claim 4 (depends on 1)

The system of claim 1, wherein the program instructions when executed cause the anomaly detection system to:
generate a plurality of host feature vectors for a plurality of hosts based on feature vectors of processes observed on respective ones of the hosts; and
process the host feature vectors using one or more other outlier detection models to determine one or more anomalous hosts among the plurality of hosts; and
output the one or more anomalous hosts.

### Claim 5 (depends on 1)

The system of claim 1, wherein the program instructions when executed cause the anomaly detection system to:
collect metadata about processes observed on the one or more hosts over successive observation periods; and
generate reports identifying anomalous processes detected by the anomaly detection system for individual ones of the observation periods.

### Claim 6 (depends on 1)

The system of claim 1, wherein:
the one or more hosts are part of a client network monitored by the anomaly detection system; and
the one or more remediation processes comprise isolating a part of the client network or making one or more configuration changes in the client network.

### Claim 7 (depends on 1)

The system of claim 1, wherein an individual observation record for an individual process indicates one or more of:
(a) a command line used to launch the individual process,
(b) a list of network addresses that the individual process connected to,
(c) one or more software modules loaded by the individual process, and
(d) one or more digital fingerprints of the one or more software modules loaded by the individual process.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more outlier detection models includes one or more of:
a K-Nearest Neighbors (KNN) clustering model,
a Random Forest model,
a Logistic Regression model,
a Support Vector Machine model, and
a Neural Network model.

### Claim 9 (depends on 1)

The system of claim 1, wherein the one or more outlier detection models are retrained periodically based on a configured schedule.

### Claim 10 (depends on 1)

The system of claim 1, wherein to extract features from an individual observation record, the the program instructions when executed cause the anomaly detection system to anomaly detection system is configured to generate Term Frequency-Inverse Document Frequency (TF-IDF) measures of individual feature values in the individual observation record.

### Claim 11

A method comprising:
performing, by one or more computing devices that implement an anomaly detection system:
receiving a plurality of observation records indicating metadata about a plurality of processes executed on one or more hosts, wherein the processes belong to a process category;
for each observation record of a process:
extracting a plurality of features of the process from the observation record to generate a feature vector; and
processing the feature vector using one or more outlier detection models to determine an outlier metric for the process within the process category, wherein the one or more outlier detection models are repeatedly retrained using one or more unsupervised machine learning techniques based on a first data store containing normal observation records of the process category;
ranking the processes based on respective outlier metrics of the processes;
detecting a set of anomalous processes among the processes, wherein the set of anomalous processes is limited to a specified number or percentile of the processes that have highest outlier metric rankings;
detecting one or more observation records of the anomalous processes in the set that indicates a signal of a potential cyberattack, wherein the signal is detected based on a scan of a second data store containing anomalous observation records associated with confirmed cyberattacks;
generating, via a graphical user interface (GUI), an alert indicating the signal; and
in response to confirmation of the signal received via the GUI:
removing the one or more observation records from the first data store used to retrain the one or more outlier detection models;
archiving the one or more observation records in the second data store as an example signal of the potential cyberattack; and
initiating one or more remediation processes to mitigate against the potential cyberattack.

### Claim 12 (depends on 11)

The method of claim 11, further comprising the anomaly detection system:
categorizing observed processes on a host into a plurality of process categories including the process category, wherein the observed processes are categorized based on one or more of:
(a) an executable used to launch the observed process,
(b) a file system path of the executable,
(c) a version of the executable,
(d) a MD5 value of the executable,
(e) a user who launched the process, and
(f) a time when the process was launched.

### Claim 13 (depends on 12)

The method of claim 12, wherein the process categories are related to one another in a hierarchical process tree.

### Claim 14 (depends on 11)

The method of claim 11, further comprising the anomaly detection system:
generating a plurality of host feature vectors for a plurality of hosts based on feature vectors of processes observed on respective ones of the hosts; and
processing the host feature vectors using one or more other outlier detection models to determine one or more anomalous hosts among the plurality of hosts; and
outputting the one or more anomalous hosts.

### Claim 15 (depends on 11)

The method of claim 11, further comprising the anomaly detection system:
collecting metadata about processes observed on the one or more hosts over successive observation periods; and
generating reports identifying anomalous processes detected by the anomaly detection system for individual ones of the observation periods.

### Claim 16 (depends on 11)

The method of claim 11, wherein:
the one or more hosts are part of a client network monitored by the anomaly detection system; and
the one or more remediation processes comprise isolating a part of the client network or making one or more configuration changes in the client network.

### Claim 17 (depends on 11)

The method of claim 11, wherein an individual observation record for an individual process indicates one or more of:
(a) a command line used to launch the individual process,
(b) a list of network addresses that the individual process connected to,
(c) one or more software modules loaded by the individual process, and
(d) one or more digital fingerprints of the one or more software modules loaded by the individual process.

### Claim 18 (depends on 11)

The method of claim 11, wherein the one or more outlier detection models includes one or more of:
a K-Nearest Neighbors (KNN) clustering model,
a Random Forest model,
a Logistic Regression model,
a Support Vector Machine model, and
a Neural Network model.

### Claim 19 (depends on 11)

The method of claim 11, wherein the one or more outlier detection models are retrained periodically based on a configured schedule.

### Claim 20 (depends on 11)

The method of claim 11, wherein generating the feature vector of the process includes reducing a number of extracted features of the process using one or more dimensionality reduction techniques including one or more of a Principal Component Analysis (PCA) technique, a Non-Negative Matrix Factorization (NMF) technique, or an Auto Encoder technique.