Independent data processing environments within a big data cluster system
AssigneeDatabricks, Inc.
InventorsAli Ghodsi, Ion Stoica
Life of this patent
Estimating the term…
Abstract
A cluster system includes an interface and a processor. The interface is to receive a request from a user associated with one of a plurality of shells. The processor is to determine a plurality of tasks to respond to the request; determine a local set of data and a shared set of data for a task of the plurality of tasks, wherein the local set of data is associated with the one of the plurality of shells; and provide the task, a local set indication, and a shared set indication to a worker associated with the task, wherein the local set indication refers to the local set of data and the shared set indication refers to the shared set of data.
Claims20
A processor configured to:
- receive a request associated with one of a plurality of shells;
- determine a local set of data and a shared set of data for a task based on the request, wherein the local set of data is accessed by a specific one of the plurality of shells such that a first variable included in a first shell represents data independent of data represented by a second variable included in a second shell and the shared set of data has a shared set indication;
- provide the task, a local set indication, and the shared set indication to a worker associated with the task, wherein the local set indication is associated with the local set of data;
- receive a task result from the worker;
- generate a job result including combining the task result with at least one other task result;
- and
- output the job result.
A method, comprising:
- receiving a request associated with one of a plurality of shells;
- determining a local set of data and a shared set of data for a task based on the request, wherein the local set of data is accessed by a specific one of the plurality of shells such that a first variable included in a first shell represents data independent of data represented by a second variable included in a second shell and the shared set of data has a shared set indication;
- providing the task, a local set indication, and the shared set indication to a worker associated with the task, wherein the local set indication is associated with the local set of data;
- receiving a task result from the worker;
- generating a job result including combining the task result with at least one other task result;
- and
- outputting the job result.
A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- receiving a request associated with one of a plurality of shells;
- determining a local set of data and a shared set of data for a task based on the request, wherein the local set of data is accessed by a specific one of the plurality of shells such that a first variable included in a first shell represents data independent of data represented by a second variable included in a second shell and the shared set of data has a shared set indication;
- providing the task, a local set indication, and the shared set indication to a worker associated with the task, wherein the local set indication is associated with the local set of data;
- receiving a task result from the worker;
- generating a job result including combining the task result with at least one other task result;
- and
- outputting the job result.
Dependencies are inferred from the claim text.