# US 10,310,903 B2 · Resilient scheduling of broker jobs for asynchronous tasks in a multi-tenant platform-as-a-service (PaaS) system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,903 B2 - Document ID: US10310903B2 - Status: Granted - Kind: B2 - Application number: 14/158,164 - Filed: 2014-01-17 - Published: 2015-07-23 - Granted: 2019-06-04 - Earliest claimed: 2014-01-17 - Assignees: Red Hat, Inc. - Applicants: Red Hat, Inc. - Inventors: Daniel McPherson; Abhishek Gupta; Jordan Liggitt - Web page: https://patentgrep.com/patent/US10310903B2 ## CPC - G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals} - G06F 9/466: {Transaction processing} - G06F 9/5038: {considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups)} - G06F 11/1474: in transactions (updating of structured data in databases G06F16/23) - G06F 2209/5013: Request control - G06F 2209/5017: Task decomposition - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Implementations for resilient scheduling of broker jobs for asynchronous tasks in a multi-tenant Platform-as-a-Service (PaaS) system are disclosed. A method of the disclosure includes receiving, by the processing device of a broker of a multi-tenant PaaS system from a user of the multi-tenant PaaS system, a request to complete a job, adding, by the processing device, an entry corresponding to the requested job in a data store of the broker, adding, by the processing device, another entry corresponding to the requested job in a scheduler communicably coupled to the broker, and sending, by the processing device to the user, an acknowledgment of the request and an identifier (ID) of the job, wherein the job is processed asynchronous to the sending of the acknowledgment. ## Claims ### Claim 1 A method, comprising: receiving, by a processing device of a broker of a multi-tenant Platform-as-a-Service (PaaS) system from a user device of the multi-tenant PaaS system, a first request to complete a job; sending, by the processing device to the user device, a processing status of the job; collecting, by the processing device, information of a plurality of nodes, applications residing on the plurality of nodes, and software components utilized by the applications residing on the plurality of nodes; generating, by the processing device, a model of the broker using the information, wherein the model represents the plurality of nodes, the applications, and the software components, wherein the model specifies a plurality of sub-operations for the job and corresponding re-execution parameters for retrying or rolling back each of the sub-operations; invoking, by a worker component of a server device of the broker, the model of the broker to elaborate the job into the plurality of sub-operations, store the plurality of sub-operations in a data store of the broker, and associate the plurality of sub-operations with the job; and executing, by the worker component as part of processing the job, each of the plurality of sub-operations transactionally, wherein first re-execution parameters of a first operation of the plurality of sub-operations comprise an indication that the first operation is to re-execute as-is without regard to a state of a previous execution attempt, wherein second re-execution parameters of a second operation of the plurality of sub-operations specify a list of earlier operations that are to be rolled back before retry along with subsequent operations that depend on the earlier operations in the list. ### Claim 2 (depends on 1) The method of claim 1, wherein the job comprises at least one of creating a new application, adding a component to an existing application, building an application, deploying an application, deleting an application, scaling up/down an application, distributing Secure Shell (SSH) keys, or distributing environment variables. ### Claim 3 (depends on 1) The method of claim 1, further comprising adding, by the processing device, an entry corresponding to the job in the data store of the broker, wherein the entry added to the data store of the broker comprises at least one of a job identifier (ID) field, a job type field, a title field, a description field, arguments, a child jobs field, a parent job field, a state field, a completion status field, a retry count field, a rollback retry count field, a percentage complete field, a result field, an object type field, an application ID field, an application name field, a domain name field, an owner login field, a creator login field, or an object Uniform Resource Locator (URL) field. ### Claim 4 (depends on 3) The method of claim 3, further comprising: adding, by the processing device, another entry corresponding to the job in a scheduler communicably coupled to the broker; and setting the state field of the entry to a ‘scheduled’ status to indicate that the job has been added to the scheduler. ### Claim 5 (depends on 1) The method of claim 1, further comprising: adding, by the processing device, an entry corresponding to the job in the data store of the broker, adding, by the processing device, another entry corresponding to the job in a scheduler communicably coupled to the broker; and identifying one or more of the plurality of sub-operations in the data store of the broker that do not have a ‘scheduled’ status in a state field to determine whether the one or more sub-operations are to be scheduled at the scheduler, wherein the identifying provides resiliency to the scheduler. ### Claim 6 (depends on 1) The method of claim 1, wherein a roll-back policy flag and a re-try policy flag of the corresponding re-execution parameters for each of the plurality of sub-operations are pre-configured and known by the worker component as part of the model of the broker. ### Claim 7 (depends on 1) The method of claim 1, wherein third re-execution parameters of a third operation of the plurality of sub-operations comprises an indication that the third operation is to be rolled back before retry. ### Claim 8 (depends on 1) The method of claim 1, further comprising: sending, by the processing device to the user device, an acknowledgment of the first request and an identifier (ID) of the job, wherein the job is processed asynchronously with respect to the sending of the acknowledgment; and receiving, by the processing device, a second request for the processing status of the job, the second request comprising the ID of the job. ### Claim 9 A system, comprising: a memory; and a processing device communicably coupled to the memory, the processing device to: receive, from a user device of a multi-tenant Platform-as-a-Service (PaaS) system, a first request to complete a job; collect information of a plurality of nodes, applications residing on the plurality of nodes, and software components utilized by the applications residing on the plurality of nodes; generate a model of a broker, wherein the model represents the plurality of nodes, the applications, and the software components, wherein the model specifies a plurality of sub-operations for the job and corresponding re-execution parameters for retrying or rolling back each of the sub-operations; invoke the model of the broker of the multi-tenant PaaS system to elaborate the job into the plurality of sub-operations; store the plurality of sub-operations to a data store of the broker, the plurality of sub-operations corresponding to the job in the data store; execute each sub-operation of the plurality of sub-operations by a worker component as part of processing of the job; complete the job when all of the plurality of the sub-operations are executed completely; and for each respective sub-operation of the plurality of sub-operations that does not execute completely, process the respective sub-operation according to the corresponding re-execution parameters corresponding to the respective sub-operation, wherein first re-execution parameters of a first operation of the plurality of sub-operations comprise an indication that the first operation is to re-execute as-is without regard to a state of a previous execution attempt, wherein second re-execution parameters of a second operation of the plurality of sub-operations specify a list of earlier operations that are to be rolled back before retry along with subsequent operations that depend on the earlier operations in the list. ### Claim 10 (depends on 9) The system of claim 9, wherein the job comprises at least one of creating a new application, adding a component to an existing application, building an application, deploying an application, deleting an application, scaling up/down an application, distributing Secure Shell (SSH) keys, or distributing environment variables. ### Claim 11 (depends on 9) The system of claim 9, wherein the processing device is further to: add a first entry corresponding to the job in the data store and a second entry corresponding to the job in a scheduler, wherein the scheduler separates processing of the job from a web request that requests completion of the job. ### Claim 12 (depends on 9) The system of claim 9, wherein, when each sub-operation of the plurality of sub-operations executes completely, the worker component is to update a job status and a job state for the job in the data store of the broker to reflect execution completion of the respective sub-operation. ### Claim 13 (depends on 9) The system of claim 9, wherein the processing device is to update a job status and a job state for the job in the data store of the broker when all of the plurality of sub-operations have executed completely. ### Claim 14 (depends on 9) The system of claim 9, wherein the processing device is further to: add a first entry corresponding to the job in the data store and a second entry corresponding to the job in a scheduler, wherein the corresponding re-execution parameters of each of the plurality of sub-operations provides resiliency to the scheduler. ### Claim 15 (depends on 9) The system of claim 9, wherein the processing device is further to: send, to the user, an acknowledgment of the first request and an identifier (ID) of the job, wherein the job is processed asynchronously with respect to sending of the acknowledgment; receive a second request for a processing status of the job, the second request comprising the ID of the job; and reserve the job from a scheduler of the multi-tenant PaaS system. ### Claim 16 A non-transitory machine-readable storage medium including instructions that, when accessed by a processing device, cause the processing device to: receive, by the processing device of a broker of a multi-tenant Platform-as-a-Service (PaaS) system from a user device of the multi-tenant PaaS system, a first request to complete a job; collect information of a plurality of nodes, applications residing on the plurality of nodes, and software components utilized by the applications residing on the plurality of nodes; generate a model of the broker using the information, wherein the model represents the plurality of nodes, the applications, and the software components, wherein the model specifies a plurality of sub-operations for the job and corresponding re-execution parameters for retrying or rolling back each of the sub-operations; invoke, by a worker component of a server device of the broker, the model of the broker to elaborate the job into the plurality of sub-operations, store the plurality of sub-operations in a data store, and associate the plurality of sub-operations with the job; and executing, by the worker component as part of processing the job, each of the plurality of sub-operations transactionally, wherein first re-execution parameters of a first operation of the plurality of sub-operations comprise an indication that the first operation is to re-execute as-is without regard to a state of a previous execution attempt, wherein second re-execution parameters of a second operation of the plurality of sub-operations specify a list of earlier operations that are to be rolled back before retry along with subsequent operations that depend on the earlier operations in the list. ### Claim 17 (depends on 16) The non-transitory machine-readable storage medium of claim 16, wherein the job comprises at least one of creating a new application, adding a component to an existing application, building an application, deploying an application, deleting an application, scaling up/down an application, distributing Secure Shell (SSH) keys, or distributing environment variables. ### Claim 18 (depends on 16) The non-transitory machine-readable storage medium of claim 16, wherein the processing device is further to add an entry corresponding to the job in the data store of the broker wherein the entry added to the data store of the broker comprises at least one of a job identifier (ID) field, a job type field, a title field, a description field, arguments, a child jobs field, a parent job field, a state field, a completion status field, a retry count field, a rollback retry count field, a percentage complete field, a result field, an object type field, an application ID field, an application name field, a domain name field, an owner login field, a creator login field, or an object Uniform Resource Locator (URL) field. ### Claim 19 (depends on 16) The non-transitory machine-readable storage medium of claim 16, wherein the processing device is further to: add an entry corresponding to the job in the data store of the broker, add another entry corresponding to the job in a scheduler communicably coupled to the broker; and identify one or more of the plurality of sub-operations in the data store of the broker that do not have a ‘scheduled’ status in a state field to determine whether the one or more sub-operations are to be scheduled at the scheduler, wherein identification of the one or more sub-operations provides resiliency to the scheduler. ### Claim 20 (depends on 16) The non-transitory machine-readable storage medium of claim 16, wherein a roll-back policy flag and a re-try policy flag of the corresponding re-execution parameters for each of the plurality of sub-operations are pre-configured and known by the worker component as part of the model of the broker. ### Claim 21 (depends on 16) The non-transitory machine-readable storage medium of claim 16, wherein the processing device is further to: send, by the processing device to the user device, an acknowledgment of the first request and an identifier (ID) of the job, wherein the job is processed asynchronously with respect to sending of the acknowledgment; and receive, by the processing device, a second request for a processing status of the job, the second request comprising the ID of the job.