Multi-tenant license enforcement across job requests

A method for processing job requests, comprising:

• receiving a stream of job requests, each job request being associated with a particular tenant of a plurality of tenants;
• persistently tracking a state of each of at least some of the job requests using a job state tracking component;
• persistently placing the job requests in a queue, wherein each of the job requests remains in the queue until the each job request has been updated to a completed state or to a cancelled state;
• reviewing a plurality of license agreements, each license agreement corresponding to at least one of the plurality of tenants;
• reviewing the job requests in the queue, each job request being associated with a particular tenant of a plurality of tenants;
• selecting a job request from the queue based on the review of the plurality of license agreements corresponding to the tenants associated with the job requests;
• creating a stream of job requests to be scheduled, the stream comprising the selected job request;
• submitting the stream of job requests to a resource manager for scheduling for processing, wherein the selected job request remains in the queue until the selected job request has been updated to a completed state or to a cancelled state;
• managing visibility of job requests during processing of the job requests by the resource manager, including setting the selected job request to an invisible state such that the selected job request is ignored during future selection of job requests to be performed while the selected job request is in the invisible state;
• and
• notifying the job state tracking component when a job request state has changed including when the job request is processing by the resource manager, when the job request is cancelled, and when the job request has been completed by the resource manager.

A system comprising:

• one or more computer processors;
• and
• data storage having encoded therein computer executable instructions which, when executed upon the one or more processors, cause the processors to instantiate:
• a queue configured to persistently store a plurality of job requests that await scheduling, each job request being associated with a particular tenant of a plurality of tenants and each job request having a state, wherein each of the plurality of job requests remains in the queue at least until it is set to a completed or a cancelled state;
• a subscriber/publisher pool;
• a plurality of job state tracking components configured to persistently track a state of each of the job requests for at least a portion of a lifetime of each of at least some of the plurality of job requests, each job state tracking component subscribing to scheduling notifications, processing notification and completion notifications from a subscriber/publisher pool, control visibility of the one or more job requests during processing of the job requests, and being configured to change the persisted state of a particular job request in response to detecting a scheduling, processing, and completion notification that corresponds to the particular job request;
• a software license agreement enforcer configured to
• review the plurality of job requests in the queue,
• review a plurality of license agreements, each license agreement corresponding to at least one of the plurality of tenants,
• select a pending job request to be performed based on the review of the plurality of license agreements corresponding to the tenants associated with the job requests, and
• provide the selected job request to a resource manager that schedules one or more compute resources that support and enable performance of the job request;
• wherein, when the selected job request is provided to the resource manager for scheduling, the job state tracking component sets the selected job request to a processing state such that the software license agreement enforcer ignores the selected job request is ignored during future selection of job requests to be performed while the selected job request in the processing state,
• wherein the selected job request remains in the queue until the selected job request has been updated to a completed state or to a cancelled state, and
• wherein the resource manager is configured to publish processing notifications into the subscriber/publisher pool when a job request begins processing by compute resources, and is configured to publish completion notifications into the subscriber/publisher pool when a job request completes processing by compute resources.