Independent resource scheduling for distributed data processing programs
AssigneeAmazon Technologies, Inc.
InventorsAnthony Virtuoso, Turkay Mert Hocanin
Life of this patent
Estimating the term…
Abstract
Independent scheduling of resources may be performed for distributed data processing programs. When a request is received that causes the performance of a distributed data processing program, a plan to perform the distributed data processing program may be generated. Different resources may be obtained to perform different portions of the distributed data processing program according to different portions of the generated plan. A result of the request may be provided to a user. In some embodiments, resources obtained for different portions of the distributed data processing program may be returned for use by other programs.
Claims20
A system, comprising:
- a memory to store program instructions which, when performed by at least one processor, cause the at least one processor to perform a method to at least:
- receive a request to cause performance of a distributed data processing program;
- generate a plan to perform the distributed data processing program;
- determine different computing resources to perform different portions of the distributed data processing program according to corresponding portions identified in the plan;
- send requests to obtain the different computing resources from one or more pools of computing resources during performance of the distributed data processing program, wherein at least one of the requests is sent when detecting completion of performance of one of the different portions of the distributed data processing program, and wherein, when detecting completion of the performance of the one of the different portions, one of the different computing resources is re-assigned for performing a next one of the different portions of the distributed data processing program according to the plan or returned to the one or more pools of computing resources;
- perform the distributed data processing program using the obtained computing resources;
- and
- provide a result of the request.
A method, comprising:
- receiving a request to cause performance of a distributed data processing program;
- generating a plan to perform the distributed data processing program;
- obtaining different computing resources to perform different portions of the distributed data processing program during performance of the distributed processing program, wherein, responsive to detecting completion of performance of one of the different portions, one of the different computing resources is re-assigned for performing a next one of the different portions of the distributed data processing program according to the plan or returned to a source for the one of the different computing resources;
- and
- causing a result of the request to be provided.
A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:
- receiving a request to cause performance of a distributed data processing program;
- generating a plan to perform the distributed data processing program;
- determining different computing resources to perform different portions of the distributed data processing program during performance of the distributed data processing program according to corresponding portions identified in the plan;
- obtaining the different computing resources to perform the distributed data processing program during performance of the distributed data processing program, wherein, responsive to detecting completion of performance of one of the different portions, one of the different computing resources is re-assigned for performing a next one of the different portions of the distributed data processing program according to the plan or returned to a source for the one of the different computing resources;
- and
- causing a result of the request to be provided.
Dependencies are inferred from the claim text.