US 10,757,177 B1 · Data locality-aware MapReduce using IoT devices

Abstract

A client network implements data locality-aware MapReduce using IoT devices. An IoT service of a provider network receives configuration settings from a user to implement map nodes and reduce nodes at the client network. Each map node is configured to receive its own locality-based partition of data from one or more data sources (e.g., sensors) that are within a map function data locality (e.g., a particular location of the network). Each map node then processes the locality-based partition of the data, generates results, and sends the results to a reduce node. The reduce node may be located at a remote provider network. A coordinator node may also be deployed to one or more devices of the client network. If a map node or a compute node becomes overloaded, the coordinator node may redirect some data to a map node or a reduce node of a remote provider network.

Claims20

Dependent claims

A system, comprising:

a plurality of connected devices, respectively comprising:
one or more processors;
and
one or more memories having stored thereon instructions, which when executed by the one or more processors, cause the one or more processors to implement a map node of a MapReduce system,
wherein one of the map nodes is configured to:
receive a locality-based partition of data from one or more data source devices that share a location with the map node;
process the locality-based partition of data;
generate one or more results based on the processing;
and
send the one or more results to a reduce node, and
wherein another of the map nodes is configured to:
receive another locality-based partition of the data from one or more other data source devices that share another location with the other map node;
process the other locality-based partition of the data;
generate one or more other results based on the processing;
and
send the one or more other results to the reduce node or a different reduce node.

13 more elements

A non-transitory computer-readable storage medium storing program instructions that, when executed by one or more computing devices for a service of a provider network, cause the one or more computing devices to implement:

receiving, at the service of the provider network, an indication of a configuration for a remote network to implement at least a portion of a MapReduce system, the configuration specifying:
a plurality of map function data localities, wherein each map function data locality specifies a relationship between a map function and one or more corresponding data source devices that provide data for the map function;
and
deploying the map function to respective connected devices of the remote network according to the configuration, wherein the map function is configured to, when implemented by individual ones of the respective connected devices:
receive a locality-based partition of data from the one or more data source devices, wherein the one or more data source devices share a location with the connected device that implements the map function and another one or more data source devices share another location with another of the connected devices that also implements the map function;
process the locality-based partition of the data;
generate one or more results based on the processing;
and
send the one or more results to a reduce node.

6 more elements

Dependencies are inferred from the claim text.

Data locality-aware MapReduce using IoT devices

Life of this patent

Data locality-aware MapReduce using IoT devices

Life of this patent

Abstract

Claims20