System and method for offloading computation to storage nodes in distributed system
AssigneeAlibaba Group Holding Limited
InventorShu Li
Life of this patent
Estimating the term…
Abstract
One embodiment described herein provides a distributed computing system. The distributed computing system can include a compute cluster comprising one or more compute nodes and a storage cluster comprising a plurality of storage nodes. A respective compute node can be configured to: receive a request for a computation task; obtain path information associated with data required by the computation task; identify at least one storage node based on the obtained path information; send at least one computation instruction associated with the computation task to the identified storage node; and receive computation results from the identified storage node subsequently to the identified storage node performing the computation task.
Claims20
A distributed computing system, the system comprising:
- a compute cluster comprising one or more compute nodes;
- a storage cluster comprising a plurality of storage nodes;
- and
- a master node for distributing data among the storage nodes;
- wherein a respective storage node comprises:
- a processor;
- a memory;
- a receiving module configured to receive, from a compute node, computation instructions associated with a computation task, wherein the computation task is divided into a number of sub-tasks in such a way that each sub-task only requires data stored on a single storage node;
- a computation module configured to execute a corresponding sub-task using data stored in the storage node without requesting additional data from a different storage node, and send a computation result to the compute node;
- a path-querying module, wherein in response to determining that executing the sub-task updates data locally stored on the storage node, the path-querying module is configured to query the master node to identify other storage nodes in the storage cluster that store replicas of the locally stored data;
- and
- an update module configured to send the updated data to the identified other storage nodes.
A computer-implemented method for offloading computation tasks from a compute cluster comprising one or more compute nodes to a storage cluster comprising a plurality of storage nodes in a distributed computing system, the method comprising:
- receiving, by a storage node from a compute node, computation instructions associated with a computation task, wherein the computation task is divided into a number of sub-tasks in such a way that each sub-task only requires data stored on a single storage node;
- executing, by the storage node, a corresponding sub-task using data stored in the storage node without requesting additional data from a different storage node;
- sending a computation result to the compute node;
- in response to determining that executing the sub-task updates data locally stored on the storage node, querying a master node in the distributed computing system to identify other storage nodes in the storage cluster that store replicas of the data;
- and
- sending the updated data to the identified other storage nodes.
A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for offloading computation tasks from a compute cluster comprising one or more compute nodes to a storage cluster comprising a plurality of storage nodes in a distributed computing system, the method comprising:
- receiving, by a storage node from a compute node, computation instructions associated with a computation task, wherein the computation task is divided into a number of sub-tasks in such a way that each sub-task only requires data stored on a single storage node;
- executing, by the storage node, a corresponding sub-task using data stored in the storage node without requesting additional data from a different storage node;
- sending a computation result to the compute node;
- in response to determining that executing the sub-task updates data locally stored on the storage node, querying a master node in the distributed computing system to identify other storage nodes in the storage cluster that store replicas of the data;
- and
- sending the updated data to the identified other storage nodes.
Dependencies are inferred from the claim text.