# US 2021/0240679 A1 · Mechanism for stream processing efficiency using probabilistic model to reduce data redundancy

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240679 A1
- Document ID: US20210240679A1
- Status: Published application
- Kind: A1
- Application number: 16/779,395
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: salesforce.com, inc.
- Applicants: Salesforce.Com, Inc.
- Inventors: Yogesh Patel; Percy Mehta; Mattia Padovani; Shan-Cheng Ho; Shaahin Mehdinezhad Rushan; Johannes Kienzle
- Web page: https://patentgrep.com/patent/US20210240679A1

## CPC

- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method and system of data deduplication for data streams in a multi-tenant system. The method receives, at a data accuracy manager, an event from an activity tracking component, determine whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify the event has not been previously received with a possible false positive response where the event has been previously received, determines whether information for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events, and discards the event in response to determining that the event is recorded in the probabilistic model and in the metric storage.

## Claims

### Claim 1

A method of data deduplication for data streams in a multi-tenant system, the method comprising:
receiving, at a data accuracy manager, an event from an activity tracking component;
determining whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify the event has not been previously received with a possible false positive response where the event has been previously received;
determining whether information for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events; and
discarding the event in response to determining that the event is recorded in the probabilistic model and in the metric storage.

### Claim 2 (depends on 1)

The method of data deduplication of claim 1, wherein determining whether the event is recorded in the probabilistic model is a check for a primary key for the event where the primary key is used for the event in the metric storage.

### Claim 3 (depends on 1)

The method of data deduplication of claim 1, wherein the probabilistic model is a bloom filter.

### Claim 4 (depends on 1)

The method of data deduplication of claim 1, further comprising:
adding a unique identifier of the event to the probabilistic model in response to failing to find the unique identifier in the probabilistic model.

### Claim 5 (depends on 1)

The method of data deduplication of claim 1, further comprising:
adding the information for the event to the metric storage in response to failing to find a primary key for the event in the probabilistic model.

### Claim 6 (depends on 1)

The method of data deduplication of claim 1, further comprising:
determining whether the probabilistic model exists in response to receiving an event message.

### Claim 7 (depends on 6)

The method of data deduplication of claim 6, further comprising:
initializing the probabilistic model with a unique identifier for the event, in response to determining that the probabilistic model does not exist.

### Claim 8 (depends on 1)

The method of data deduplication of claim 1, wherein the event is generated by an activity tracker.

### Claim 9

A non-transitory machine-readable storage medium that provides instructions that, if executed by a processor, will cause said processor to perform operations comprising:
receiving, at a data accuracy manager, an event from an activity tracking component;
determining whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify the event has not been previously received with a possible false positive response where the event has been previously received;
determining whether information for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events; and
discarding the event in response to determining that the event is recorded in the probabilistic model and in the metric storage.

### Claim 10 (depends on 9)

The non-transitory machine-readable storage medium of claim 9, wherein determining whether the event is recorded in the probabilistic model is a check for a primary key for the event where the primary key is used for the event in the metric storage.

### Claim 11 (depends on 9)

The non-transitory machine-readable storage medium of claim 9, wherein the probabilistic model is a bloom filter.

### Claim 12 (depends on 9)

The non-transitory machine-readable storage medium of claim 9, the operations further comprising:
adding a unique identifier of the event to the probabilistic model in response to failing to find the unique identifier in the probabilistic model.

### Claim 13 (depends on 9)

The non-transitory machine-readable storage medium of claim 9, the operations further comprising:
adding the information for the event to the metric storage in response to failing to find a primary key for the event in the probabilistic model.

### Claim 14 (depends on 9)

The non-transitory machine-readable storage medium of claim 9, the operations further comprising:
determining whether the probabilistic model exists in response to receiving an event message.

### Claim 15 (depends on 14)

The non-transitory machine-readable storage medium of claim 14, the operations further comprising:
initializing the probabilistic model with a unique identifier for the event, in response to determining that the probabilistic model does not exist.

### Claim 16 (depends on 10)

The non-transitory machine-readable storage medium of claim 10, wherein the event is generated by an activity tracker.

### Claim 17

A computing device to execute a method of data deduplication for data streams in a multi-tenant system, the computing device comprising:
a non-transitory machine-readable medium having stored therein a data accuracy manager; and
a processor coupled to the non-transitory machine-readable medium, the processor to execute the data accuracy manager, the data accuracy manager to receive an event from an activity tracking component, determine whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify the event has not been previously received with a possible false positive response where the event has been previously received, to determine whether information for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events, and to discard the event in response to determining that the event is recorded in the probabilistic model and in the metric storage.

### Claim 18 (depends on 17)

The computing device of claim 17, wherein determining whether the event is recorded in the probabilistic model is a check for a primary key for the event where the primary key is used for the event in the metric storage.

### Claim 19 (depends on 17)

The computing device of claim 17, wherein the probabilistic model is a bloom filter.

### Claim 20 (depends on 17)

The computing device of claim 17, wherein the data accuracy manager is further to add a unique identifier of the event to the probabilistic model in response to failing to find the unique identifier in the probabilistic model.
