Hardware acceleration of processing using decentralized processing unit systems

A system comprising:

• a storage configured to store instructions;
• and
• a hardware processor configured to execute the instructions and cause the processor to:
• receive, by a dedicated piece of hardware on a network from one or more network devices though a custom pipeline to an edge of the network, telemetry data from the one or more network devices for pre-processing prior to transmission to a cloud-based server, wherein the telemetry data is offloaded from other devices on the network to the dedicated piece of hardware and the dedicated piece of hardware is configured for hardware acceleration, wherein the custom pipeline is configured to route the telemetry data from the one or more network devices to the dedicated piece of hardware via a routing path that bypasses or is separate from a general data forwarding path;
• identify, by the dedicated piece of hardware, duplicative records within the telemetry data;
• generate, by the dedicated piece of hardware, a reduced set of telemetry data that has the duplicative records removed;
• process, by the dedicated piece of hardware, the reduced set of telemetry data into a processed dataset;
• and
• send, by the dedicated piece of hardware, the processed dataset to the cloud-based server over the network.

A non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to:

• receive, by a dedicated piece of hardware on a network from one or more network devices though a custom pipeline to an edge of the network, telemetry data from the one or more network devices for pre-processing prior to transmission to a cloud-based server, wherein the telemetry data is offloaded from other devices on the network to the dedicated piece of hardware and the dedicated piece of hardware is configured for hardware acceleration, wherein the custom pipeline is configured to route the telemetry data from the one or more network devices to the dedicated piece of hardware via a routing path that bypasses or is separate from a general data forwarding path;
• identify, by the dedicated piece of hardware, duplicative records within the telemetry data;
• generate, by the dedicated piece of hardware, a reduced set of telemetry data that has the duplicative records removed;
• process, by the dedicated piece of hardware, the reduced set of telemetry data into a processed dataset;
• and
• send, by the dedicated piece of hardware, the processed dataset to the cloud-based server over the network.