Extensible, secure and efficient monitoring & diagnostic pipeline for hybrid cloud architecture

A method implemented by an on-premises computing system for performing event storage and diagnostic processing in a hybrid cloud environment that includes a cloud storage and a cloud file sync service, the on-premises computing system comprising one or more processors and one or more computer-readable media having stored instructions that are executable by the one or more processors to implement the method, wherein the method includes the on-premises computing system performing at least the following:

• identifying a plurality of different monitoring events of one or more machines in the on-premises computing system during a predetermined batch period of time;
• generating a plurality of event records corresponding to the plurality of different monitoring events detected during the predetermined batch period of time, wherein generating the plurality of event records includes determining a format to configure each event record with, selectively based on a type of monitoring event associated with each event record, and wherein the plurality of event records include at least two different event record format types;
• generating appended event records by appending the event records with correlation vector data that identifies, for each event record, a context associated with the monitored event that is associated with the event record;
• generating a batch of event records by batching all of the appended event records that correspond to the different monitoring events that were identified during the predetermined batch period of time;
• obtain a security key from the cloud file sync service that is uniquely associated with a corresponding security key provided to the cloud storage for restricting access to a particular storage container associated with the on-premises computing system and that is managed by the cloud storage;
• signing the batch of event records with the security key;
• and
• transmitting the batch of event records with the security key to the cloud storage entity and which causes the batch of event records to be stored in the particular storage container.

A computer program product comprising one more hardware storage device having stored instructions that are executable by the one or more processors of a computing system for causing the computing system to implement a method that includes the computing system performing the following:

• identifying a plurality of different monitoring events of one or more machines in the computing system during a predetermined batch period of time;
• generating a plurality of event records corresponding to the plurality of different monitoring events detected during the predetermined batch period of time, wherein generating the plurality of event records includes determining a format to configure each event record with, selectively based on a type of monitoring event associated with each event record, and wherein the plurality of event records include at least two different event record format types;
• generating appended event records by appending the event records with correlation vector data that identifies, for each event record, a context associated with the monitored event that is associated with the event record;
• generating a batch of event records by batching all of the appended event records that correspond to the different monitoring events that were identified during the predetermined batch period of time;
• obtain a first security key from the cloud file sync service that is uniquely associated with a corresponding security key used to restrict access to a particular storage container associated with the computing system;
• signing the batch of event records with the security key;
• and
• transmitting the batch of event records with the security key for storage in the particular storage container.