# US 11,030,258 B1 · Ranking anomalies associated with populations of users based on relevance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,030,258 B1
- Document ID: US11030258B1
- Status: Granted
- Kind: B1
- Application number: 15/874,811
- Filed: 2018-01-18
- Granted: 2021-06-08
- Earliest claimed: 2017-01-18
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Chih Po Wen; Goutham Kurra
- Web page: https://patentgrep.com/patent/US11030258B1

## CPC

- G06N 20/20: Ensemble learning
- G06F 9/451: Execution arrangements for user interfaces
- G06F 16/2457: with adaptation to user needs
- G06F 16/24578: {using ranking}
- G06F 16/9038: Presentation of query results
- G06F 16/907: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G06F 2201/80: Database-specific techniques
- G06F 2201/875: Monitoring of systems including the internet
- G06F 2216/03: Data mining
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A system analyzes periodically collected data associated with entities, for example, users, servers, or systems. The system determines anomalies associated with populations of entities. The system excludes anomalies from consideration to increase efficiency of execution. The system may rank the anomalies based on relevance scores. The system determines relevance scores based on various factors describing the sets of entities. The system may present information describing the anomalies based on the ranking. The system may use a machine learning based model for predicting likelihoods of outcomes associated with sets of entities. The system generates alerts for reporting the outcomes based on the predictions.

## Claims

### Claim 1

A method for generating an alert, the method comprising:
storing in a database, a plurality of records of a plurality of entities, each record storing one or more attributes describing an entity of the plurality of entities;
performing data collection, the collected data representing responses associated with the plurality of entities;
identifying a set of anomalies based on the collected data, wherein each anomaly in the set of anomalies is associated with multiple entities comprising a subset of the plurality of entities;
for each anomaly in the set of anomalies, determining a relevance score for said each anomaly based on factors comprising information describing the subset of entities associated with said each anomaly, wherein the factors used for determining the relevance score for said each anomaly comprise a number of entities in the subset of entities associated with said each anomaly;
ranking the set of anomalies based on the relevance score for each anomaly in the set of anomalies;
determining a subset of the set of anomalies based on the ranking; and
sending one or more alerts based on the subset of anomalies.

### Claim 2 (depends on 1)

The method of claim 1, wherein a first anomaly associated with a first subset of entities is ranked higher than a second anomaly associated with a second subset of entities if the number of entities in the first subset entities is greater than the number of entities in the second subset of entities.

### Claim 3 (depends on 1)

The method of claim 1, wherein the factors used for determining the relevance score for said each anomaly comprise a category of anomaly based on a type of information indicated by the anomaly, the type of information representing whether the anomaly indicates negative information associated with a population.

### Claim 4 (depends on 3)

The method of claim 3, wherein a first anomaly associated with a first set of records is ranked higher than a second anomaly if the first anomaly is indicative of negative information associated with entities associated with the first set of records and the second anomaly is indicative of a positive information associated with the entities associated with the second set of records.

### Claim 5 (depends on 1)

The method of claim 1, wherein the data collection received values for a set of collected attributes, the set of collected attribute comprising an outcome attribute, and an anomaly is associated with a collected attribute, wherein the factors used for determining the relevance score for the anomaly comprise a measure of correlation between the collected attribute and the outcome attribute.

### Claim 6 (depends on 5)

The method of claim 5, wherein a first anomaly associated with a first collected attribute is ranked higher than a second anomaly associated with a second collected attribute if the first collected attribute is determined to have higher correlation with the outcome attribute compared to the second collected attribute.

### Claim 7 (depends on 1)

The method of claim 1, wherein an entity represents a user and a record stores user profile attributes.

### Claim 8 (depends on 7)

The method of claim 7, wherein performing data collection comprises presenting a set of questions to users via a user interface and receiving responses to the questions from the users.

### Claim 9 (depends on 1)

The method of claim 1, wherein an anomaly is associated with a collected attribute and the factors used for determining the relevance score for the anomaly comprise a difference between an aggregate value of the collected attribute and a baseline value of the collected attribute.

### Claim 10 (depends on 9)

The method of claim 9, wherein the baseline value of the collected attribute represents a benchmark.

### Claim 11 (depends on 9)

The method of claim 9, wherein the baseline value of the collected attribute a previous aggregate value based on the collected attribute values obtained during a previous data collection.

### Claim 12 (depends on 1)

The method of claim 1, wherein an anomaly based on an increase in an overall response rate associated with a data collection is ranked higher than an anomaly representing difference between values of a specific collected attribute and a baseline associated with the collected attribute.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
storing a machine-learned model that predicts a value of a particular metric describing a population based on input describing the population;
for each anomaly in the set of anomalies:
identifying a plurality of feature values associated with the multiple entities of said each anomaly;
using the machine-learned model to generate a particular metric based on the plurality of features values.

### Claim 14

A method comprising:
storing in a database, a plurality of records of a plurality of entities, each record storing one or more attributes describing an entity of the plurality of entities;
performing data collection, the collected data representing responses associated with the plurality of entities;
identifying a set of anomalies based on the collected data, wherein each anomaly in the set of anomalies is associated with a set of users representing a population;
for each anomaly in the set of anomalies, determining a relevance score for the anomaly;
ranking the set of anomalies based on the relevance scores, wherein ranking the set of anomalies comprises:
determining that a first anomaly is associated with a first population that corresponds to a pre-defined team;
determining that a second anomaly is associated with a second population that does not correspond to any pre-defined team;
ranking the first anomaly higher than the second anomaly at least based in part on the first population corresponding to the pre-define team and the second population not corresponding to any pre-defined team;
determining a subset of anomalies based on the ranking; and
sending one or more alerts based on the subset of anomalies.

### Claim 15

A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause:
storing, in a database, a plurality of records of a plurality of entities, each record storing one or more attributes describing an entity of the plurality of entities;
performing data collection, the collected data representing responses associated with the plurality of entities;
identifying a set of anomalies based on the collected data, wherein each anomaly in the set of anomalies is associated with multiple entities comprising a subset of the plurality of entities;
for each anomaly in the set of anomalies, determining a relevance score for said each anomaly based on factors comprising information describing the subset of entities associated with said each anomaly, wherein the factors used for determining the relevance score for said each anomaly comprise a number of entities in the subset of entities associated with said each anomaly;
ranking the set of anomalies based on the relevance score for each anomaly in the set of anomalies;
determining a subset of the set of anomalies based on the ranking; and
sending one or more alerts based on the subset of anomalies.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the data collection received values for a set of collected attributes, the set of collected attribute comprising an outcome attribute, and an anomaly is associated with a collected attribute, wherein the factors used for determining the relevance score for the anomaly comprise a measure of correlation between the collected attribute and the outcome attribute.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein a first anomaly associated with a first collected attribute is ranked higher than a second anomaly associated with a second collected attribute if the first collected attribute is determined to have higher correlation with the outcome attribute compared to the second collected attribute.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein:
the factors used for determining the relevance score for said each anomaly comprise a number of entities in the subset of entities associated with the anomaly;
a first anomaly associated with a first subset of entities is ranked higher than a second anomaly associated with a second subset of entities if the number of entities in the first subset entities is greater than the number of entities in the second subset of entities.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein:
the factors used for determining the relevance score for said each anomaly comprise a category of anomaly based on a type of information indicated by the anomaly, the type of information representing whether the anomaly indicates negative information associated with a population;
a first anomaly associated with a first subset of entities is ranked higher than a second anomaly associated with a second subset of entities if the first anomaly is indicative of negative information associated the first subset of entities and the second anomaly is indicative of a positive information associated with the second subset of entities.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, further cause:
storing a machine-learned model that predicts a value of a particular metric describing a population based on input describing the population;
for each anomaly in the set of anomalies:
identifying a plurality of feature values associated with the multiple entities of said each anomaly;
using the machine-learned model to generate a particular metric based on the plurality of features values.