Mechanism for stream processing efficiency using probabilistic model to reduce data redundancy

A computing device to execute a method of data deduplication for data streams in a multi-tenant system, the computing device comprising:

• a non-transitory machine-readable medium having stored therein a data accuracy manager;
• and
• a processor coupled to the non-transitory machine-readable medium, the processor to execute the data accuracy manager, the data accuracy manager to receive an event from an activity tracking component, determine whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify the event has not been previously received with a possible false positive response where the event has been previously received, to determine whether information for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events, and to discard the event in response to determining that the event is recorded in the probabilistic model and in the metric storage.