Secure and scalable data ingestion pipeline
AssigneePlex Systems, Inc.
InventorsPradeep Padmakar Surale, Rahul Borkar
Life of this patent
Estimating the term…
Abstract
A system and method for a secure and scalable data collection and ingestion data pipeline is discussed. The data pipeline provides secure ingestion of data into internet-of-things (IoT) platform, while providing high throughput, replication, and fault-tolerance, which makes it suitable for large scale systems where data can be received from millions or billions of devices, sensors, or clients. In addition, the data pipeline provides a platform for real time event generation and data analysis. Further the data pipeline reduces operational costs and platform-as-a-service (PAAS) vendor dependencies.
Claims20
A method for securely ingesting data from a plurality of devices over one or more networks, the method comprising:
- (a) establishing, by a data pipeline platform executing on one or more servers, a secure tunnel connection with a plurality of devices, each of the of the plurality of devices assigned by the data pipeline platform a unique application programming interface (API) key;
- (b) receiving, by the data pipeline platform, an application layer request over the secure tunnel connection via one or more networks from a device of the plurality of devices, the application layer request comprising an encrypted message from the device;
- (c) determining, by the data pipeline platform, that the application layer request comprises the unique API key assigned to the device in a predetermined authentication header of the application layer request;
- (d) accepting, by the data pipeline platform, responsive to the determination, the application layer request;
- (e) using, by the data pipeline platform responsive to accepting the request, the unique API key of the device to determine an encryption key to decrypt the encrypted message to a message;
- and
- (f) storing, by the data pipeline platform responsive to accepting the application layer request, the message to a first partition of a plurality of partitions corresponding to an input data topic.
A system for securely ingesting data from a plurality of devices over one or more networks, the system comprising:
- a data pipeline platform executing on one or more servers, wherein the data pipeline platform is configured to:
- establish a secure tunnel connection with a plurality of devices, each of the of the plurality of devices assigned by the data pipeline platform a unique application programming interface (API) key;
- receive an application layer request over the secure tunnel connection via one or more networks from a device of the plurality of devices, the application layer request comprising an encrypted message from the device;
- determine that the application layer request comprises the unique API key assigned to the device in a predetermined authentication header of the application layer request;
- accept the application layer request responsive to the determination;
- use the unique API key of the device to determine, responsive to accepting the application layer request, an encryption key to decrypt the encrypted message to a message;
- and
- store, responsive to accepting the application layer request, the message to a first partition of a plurality of partitions corresponding to an input data topic.
Dependencies are inferred from the claim text.