Back to results
US 2015/0149503 A1Published application
Parallel access to data in a distributed file system
AssigneeAb Initio Technology LLC
InventorsAnn M. Wollrath, Bryan Phil Douros, Marshall Alan Isman, Timothy Wakeling
Life of this patent
Estimating the term…
Abstract
An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.
Claims24
A method for processing data, the method including:
- receiving a specification of one or more named units stored in a distributed filesystem of a distributed processing system;
- receiving a specification for establishing data connections to a computation system separate from the distributed processing system;
- invoking a first plurality of processes on the distributed processing system, each process of the first plurality of processes establishing a data connection with a storage element of the distributed filesystem for accessing a corresponding part of the named unit in the distributed filesystem;
- using the specification for establishing the data connections to form a plurality of data connections between the distributed processing system and the computation system, at least one data connection being formed between each process of the first plurality of processes and the computation system;
- and
- passing data concurrently over the plurality of data connections from the distributed processing system to the computation system.
3 more elements
Software stored on a non-transitory computer-readable medium, for processing data, the software including instructions for causing a system to:
- receive a specification of one or more named units stored in a distributed filesystem of a distributed processing system;
- receive a specification for establishing data connections to a computation system separate from the distributed processing system;
- invoke a first plurality of processes on the distributed processing system, each process of the first plurality of processes establishing a data connection with a storage element of the distributed filesystem for accessing a corresponding part of the named unit in the distributed filesystem;
- use the specification for establishing the data connections to form a plurality of data connections between the distributed processing system and the computation system, at least one data connection being formed between each process of the first plurality of processes and the computation system;
- and
- pass data concurrently over the plurality of data connections from the distributed processing system to the computation system.
3 more elements
A system for processing data, the system including:
- a distributed processing system that includes a distributed filesystem;
- and
- a computation system separate from the distributed processing system;
- wherein the distributed processing system is configured to:
- receive a specification of one or more named units stored in the distributed filesystem;
- receive a specification for establishing data connections to the computation system;
- invoke a first plurality of processes, each process of the first plurality of processes establishing a data connection with a storage element of the distributed filesystem for accessing a corresponding part of the named unit in the distributed filesystem;
- use the specification for establishing the data connections to form a plurality of data connections between the distributed processing system and the computation system, at least one data connection being formed between each process of the first plurality of processes and the computation system;
- and
- pass data concurrently over the plurality of data connections to the computation system.
7 more elements
A method for processing data, the method including:
- providing a specification of one or more named units stored in a distributed filesystem;
- providing a specification for establishing data connections with a computation system;
- providing a specification for processes of a first plurality of processes for invocation on a distributed processing system, each process of the first plurality of processes being specified for establishing a data connection with a storage element of the distributed filesystem for accessing a corresponding part of the named unit in the distributed filesystem;
- receiving requests to form a plurality of data connections between the distributed processing system and the computation system, and providing information for forming at least one data connection being between each process of the first plurality of processes and the computation system;
- and
- receiving data concurrently over the plurality of data connections from the first plurality of processes at the computation system.
3 more elements
Software stored on a non-transitory computer-readable medium, for processing data, the software including instructions for causing a system to:
- provide a specification of one or more named units stored in a distributed filesystem;
- provide a specification for establishing data connections with a computation system;
- provide a specification for processes of a first plurality of processes for invocation on a distributed processing system, each process of the first plurality of processes being specified for establishing a data connection with a storage element of the distributed filesystem for accessing a corresponding part of the named unit in the distributed filesystem;
- receive requests to form a plurality of data connections between the distributed processing system and the computation system, and provide information for forming at least one data connection being between each process of the first plurality of processes and the computation system;
- and
- receive data concurrently over the plurality of data connections from the first plurality of processes at the computation system.
3 more elements
A system for processing data, the system including:
- a distributed filesystem;
- a distributed processing system;
- a computation system;
- and
- a client of the distributed processing system configured to:
- provide a specification of one or more named units stored in the distributed filesystem;
- provide a specification for establishing data connections with the computation system;
- provide a specification for processes of a first plurality of processes for invocation on the distributed processing system, each process of the first plurality of processes being specified for establishing a data connection with a storage element of the distributed filesystem for accessing a corresponding part of the named unit in the distributed filesystem;
- receive requests to form a plurality of data connections between the distributed processing system and the computation system, and provide information for forming at least one data connection being between each process of the first plurality of processes and the computation system;
- and
- receive data concurrently over the plurality of data connections from the first plurality of processes at the computation system.
8 more elements
A method for processing data, the data being provided from a distributed processing system implementing a map-reduce data processing framework, the method including:
- providing to the distributed processing system a specification for a map procedure for invocation on the distributed processing system, the specification for the map procedure identifying one or more named units in a distributed filesystem for processing and including a specification for establishing data connections with a computation system separate from the distributed processing system;
- causing execution of a plurality of instances of the map procedure on the distributed processing system;
- receiving requests to form a plurality of data flow connections between executing instances of the map procedure and the computation system, and providing information for forming at least one data flow connection being between each executing instance of the map procedure and the computation system;
- and
- receiving data concurrently over the plurality of data flow connections and processing the received data in the computation system.
2 more elements
Software stored on a non-transitory computer-readable medium, for processing data, the data being provided from a distributed processing system implementing a map-reduce data processing framework, the software including instructions for causing a system to:
- provide to the distributed processing system a specification for a map procedure for invocation on the distributed processing system, the specification for the map procedure identifying one or more named units in a distributed filesystem for processing and including a specification for establishing data connections with a computation system separate from the distributed processing system;
- cause execution of a plurality of instances of the map procedure on the distributed processing system;
- receive requests to form a plurality of data flow connections between executing instances of the map procedure and the computation system, and provide information for forming at least one data flow connection being between each executing instance of the map procedure and the computation system;
- and
- receive data concurrently over the plurality of data flow connections and process the received data in the computation system.
2 more elements
A system for processing data, the system including:
- a distributed filesystem;
- a distributed processing system;
- a computation system separate from the distributed processing system;
- and
- a client of the distributed processing system configured to:
- provide to the distributed processing system a specification for a map procedure for invocation on the distributed processing system, the specification for the map procedure identifying one or more named units in the distributed filesystem for processing and including a specification for establishing data connections with the computation system;
- cause execution of a plurality of instances of the map procedure on the distributed processing system;
- receive requests to form a plurality of data flow connections between executing instances of the map procedure and the computation system, and provide information for forming at least one data flow connection being between each executing instance of the map procedure and the computation system;
- and
- receive data concurrently over the plurality of data flow connections and processing the received data in the computation system.
7 more elements
Dependencies are inferred from the claim text.