Resource de-duplicator for data-provider agnostic cloud security tool

A method for de-duplicating resource collection during policy execution, the method comprising:

• determining that a policy is to be executed;
• retrieving rules, resource identifiers, and data provider identifiers associated with the policy;
• generating a pending rules data structure and populating the pending rules data structure with identifiers of the retrieved rules and their respective necessary resources as indicated by the resource identifiers, the pending rules data structure indicating a set of the rules that has not yet been executed;
• generating a pending resources data structure with the resource identifiers and their respective data providers as indicated by the data provider identifiers, the pending resources data structure indicating a set of the resource identifiers corresponding to unretrieved resources that are to be retrieved in order to execute one or more of the rules;
• collecting, asynchronously, each resource from its associated data provider;
• as each respective resource is collected, deleting the respective resource from the pending resources data structure, and populating a collected resources data structure with an identifier of the respective resource;
• determining, as each resource is collected, whether the collected resources data structure includes identifiers of all resources needed to execute a rule in the pending rules data structure;
• executing individual ones of the retrieved rules as it is determined that their corresponding resources have identifiers populated in the collected resources data structure;
• and
• generating one or more alerts based on results obtained from the executed rules.

A system comprising:

• memory with instructions encoded thereon for de-duplicating resource collection during policy execution;
• and
• one or more processors that, when executing the instructions, are caused to perform operations comprising:
• determining that a policy is to be executed;
• retrieving rules, resource identifiers, and data provider identifiers associated with the policy;
• generating a pending rules data structure and populating the pending rules data structure with identifiers of the retrieved rules and their respective necessary resources as indicated by the resource identifiers, the pending rules data structure indicating a set of the rules that has not yet been executed;
• generating a pending resources data structure with the resource identifiers and their respective data providers as indicated by the data provider identifiers, the pending resources data structure indicating a set of the resource identifiers corresponding to unretrieved resources that are to be retrieved in order to execute one or more of the rules;
• collecting, asynchronously, each resource from its associated data provider;
• as each respective resource is collected, deleting the respective resource from the pending resources data structure, and populating a collected resources data structure with an identifier of the respective resource;
• determining, as each resource is collected, whether the collected resources data structure includes identifiers of all resources needed to execute a rule in the pending rules data structure;
• executing individual ones of the retrieved rules as it is determined that their corresponding resources have identifiers populated in the collected resources data structure;
• and
• generating one or more alerts based on results obtained from the executed rules.