Resilient data processing pipeline architecture

A method for processing plurality of data packets in a processing pipeline, comprising:

• a) receiving a plurality of declarative and defined data packets;
• b) directing each data packet to one of a plurality of collectors;
• c) verifying that each data packet is authorized to access the private cloud-based processing pipeline with one or more collectors in the plurality;
• d) delivering each of the data packets to a particular extract transform load (ETL) frontend in a plurality of ETL frontends;
• e) converting the data packets from a platform dependent form to a platform independent form with the particular ETL frontend to produce converted data packets;
• f) delivering the converted data packets to a queue;
• and
• g) processing each of the converted data packets with at least one handler of a plurality of handlers.

A nontransitory computer readable medium containing program instructions for optical switching, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out a method for processing plurality of data packets in a processing pipeline, the method comprising:

• a) receiving a plurality of declarative and defined data packets;
• b) directing each data packet to one of a plurality of collectors;
• c) verifying that each data packet is authorized to access the private cloud-based processing pipeline with one or more collectors in the plurality;
• d) delivering each of the data packets to a particular extract transform load (ETL) frontend in a plurality of ETL frontends;
• e) converting the data packets from a platform dependent form to a platform independent form with the particular ETL frontend to produce converted data packets;
• f) delivering the converted data packets to a queue;
• and
• g) processing each of the converted data packets with at least one handler of a plurality of handlers.