# US 12,007,865 B2 · Machine learning for rule evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,007,865 B2
- Document ID: US12007865B2
- Status: Granted
- Kind: B2
- Application number: 17/810,167
- Filed: 2022-06-30
- Published: 2023-10-19
- Granted: 2024-06-11
- Earliest claimed: 2022-04-16
- Assignees: Juniper Networks, Inc.
- Applicants: Juniper Networks, Inc.
- Inventors: Raja Kommula; Ganesh Byagoti Matad Sunkada; Prashanth K; Thayumanavan Sridhar; Raj Yavatkar
- Web page: https://patentgrep.com/patent/US12007865B2

## CPC

- G06F 11/3409: {for performance assessment}
- G06F 11/327: {Alarm or error message display}
- H04L 41/0604: using filtering, e.g. reduction of information by using priority, element types, position or time
- H04L 41/0609: {based on severity or priority}
- H04L 41/0681: Configuration of triggering conditions
- H04L 41/16: using machine learning or artificial intelligence
- H04L 43/024: by adaptive sampling
- H04L 43/04: Processing captured monitoring data, e.g. for logfile generation
- H04L 43/08: Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L 41/147: for predicting network behaviour
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A performance monitoring system includes a metric collector configured to receive, via metric exporters, telemetry data comprising metrics related to a network of computing devices. A metric time series database stores related metrics. An alert rule evaluator service is configured to evaluate rules using stored metrics. The performance monitoring system may include a machine learning module and is configured to determine optimized metric collection sampling intervals and rule evaluation intervals, and to automatically determine recommended alert rules.

## Claims

### Claim 1

A method comprising:
collecting, by a performance monitoring system, telemetry data comprising metrics related to a network of computing devices, wherein, for each metric, metric values associated with a corresponding metric name are collected at each of a plurality of times;
evaluating, by the performance monitoring system, alert rules using the collected telemetry data, wherein evaluating a first rule includes comparing metric values associated with a corresponding metric name of the first rule to a corresponding threshold value of the first rule at each of a plurality of rule evaluation times based on a first evaluation interval to generate a rule evaluation attribute;
determining, by the performance monitoring system, a predicted rule weight for the first rule based on the rule evaluation attribute; and
determining, by the performance monitoring system, a second evaluation interval for the first rule based on the predicted rule weight and the first evaluation interval.

### Claim 2 (depends on 1)

The method of claim 1, wherein the second evaluation interval is different than the first evaluation interval, the method further comprising subsequently evaluating, by the performance monitoring system, the first rule using the second evaluation interval.

### Claim 3 (depends on 2)

The method of claim 2, further comprising determining, by the performance monitoring system, whether the first rule is a critical rule, and if so, using a predetermined minimum evaluation interval for the second evaluation interval.

### Claim 4 (depends on 3)

The method of claim 3, wherein the rule evaluation attribute comprises a first rule evaluation attribute and a second rule evaluation attribute, wherein the first rule evaluation attribute is a rule hit rate and the second rule evaluation attribute is a rule close missed rate.

### Claim 5 (depends on 4)

The method of claim 4, further comprising determining a related rule that is related to the first rule and determining a third rule evaluation attribute that is a related rule hit rate, wherein the related rule hit rate is defined as a ratio of a number of rule hits for the related rule to a total number of rule evaluations performed for the related rule, and using the first, the second, and the third rule evaluation attributes in determining the predicted rule weight.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the predicted rule weight comprises performing a regression analysis.

### Claim 7 (depends on 1)

The method of claim 1, further comprising updating a current evaluation interval for the first rule on an on-going basis.

### Claim 8 (depends on 1)

The method of claim 1, further comprising coordinating a collection rate of the telemetry data with the second evaluation interval.

### Claim 9

A performance monitoring system, comprising:
a memory; and
one or more processors in communication with the memory, the one or more processors configured to execute a collector and an alert rule evaluator service,
wherein the collector is configured to receive telemetry data via metric exporters, the telemetry data comprising metrics related to a network of computing devices, and wherein, for each metric, metric values associated with a corresponding metric name are collected at each of a plurality of collection times, and
wherein the alert rule evaluator service is configured to evaluate rules using the received telemetry data, wherein, to evaluate a first rule, the alert rule evaluator service uses metric values associated with a corresponding metric name of the first rule, compares a corresponding metric value to a corresponding threshold value of the first rule at each of a plurality of rule evaluation times based on a first evaluation interval to generate a rule evaluation attribute, determines a predicted rule weight for the first rule based on the rule evaluation attribute, and determines a second evaluation interval for the first rule based on the predicted rule weight and the first evaluation interval.

### Claim 10 (depends on 9)

The performance monitoring system of claim 9, wherein the alert rule evaluator service subsequently evaluates the first rule using the second evaluation interval.

### Claim 11 (depends on 9)

The performance monitoring system of claim 9, wherein the alert rule evaluator service comprises a machine learning model that is trained using historical rule evaluation results.

### Claim 12 (depends on 9)

The performance monitoring system of claim 9, wherein the alert rule evaluator service determines whether the first rule is a critical rule, and if the alert rule evaluator service determines that the first rule is a critical rule, then the alert rule evaluator service sets the second evaluation interval to a predetermined minimum evaluation interval.

### Claim 13 (depends on 9)

The performance monitoring system of claim 9, wherein the rule evaluation attribute is a rule hit rate, defined as a ratio of a number of rule hits for the first rule to a total number of rule evaluations performed for the first rule.

### Claim 14 (depends on 9)

The performance monitoring system of claim 9, wherein the rule evaluation attribute includes a rule hit rate as a first rule evaluation attribute and a rule close missed rate as a second rule evaluation attribute.

### Claim 15 (depends on 14)

The performance monitoring system of claim 14, wherein the alert rule evaluator service determines a related rule that is related to the first rule, wherein the rule evaluation attribute include a third rule evaluation attribute that is a related rule hit rate, the related rule hit rate defined as a ratio of a number of rule hits for the related rule to a total number of rule evaluations performed for the related rule.

### Claim 16 (depends on 15)

The performance monitoring system of claim 15, wherein the alert rule evaluator service determines the predicted rule weight using the rule evaluation attributes and a regression analysis.

### Claim 17 (depends on 9)

The performance monitoring system of claim 9, wherein the alert rule evaluator service determines a collection rate for the telemetry data that is coordinated with the second evaluation interval.

### Claim 18 (depends on 9)

The performance monitoring system of claim 9, wherein the memory comprises a metric time series database for storing and aggregating the received telemetry data.

### Claim 19 (depends on 9)

The performance monitoring system of claim 9, wherein the first evaluation interval is a predetermined default evaluation interval.

### Claim 20 (depends on 9)

The performance monitoring system of claim 9, wherein an evaluation interval for the first rule is evaluated and updated on an on-going basis.