Techniques for fast loading of data from an external distributed file system to a database management system
AssigneeTeradata US, Inc.
InventorYu Xu
Life of this patent
Estimating the term…
Abstract
Techniques for fast loading data from an external distributed file system (DFS) to a database management system (DBMS) are provided. Assigning a portion of data from a file located in the DFS to each of a plurality of DFS clients. Configuring each DFS client to interact with a specific gateway from a plurality of gateways associated with the DBMS. Each gateway configured to send its portion of the file received from its designated client to every access module processor (AMP) controlled by that gateway. Each AMP processes a portion of the data received from the gateway to load the data into a table of the DBMS.
Claims20
A method implemented and programmed within a non-transitory computer-readable storage medium and processed by one or more processors, the processors configured to execute the method, comprising:
- configuring a first number of clients, each client configured to acquire a specific portion of data from an external distributed file system (DFS) file and interact with a specific gateway of a database management system (DBMS);
- configuring gateways of the DBMS, each of the specific gateways configured to pass the specific portion of data to every access module processor (AMP) controlled by that specific gateway;
- and
- configuring each AMP to load a unique piece of the specific portion of data that it was passed to a table of the DBMS.
A method implemented and programmed within a non-transitory computer-readable storage medium and processed by one or more processors, the processors configured to execute the method, comprising:
- initiating a predefined number of clients and directing each client to obtain a unique portion of data from a file located on an external distributed file system;
- directing each client to deliver its unique portion of the data to a specific unique gateway of a database management system (DBMS);
- instructing each gateway to simultaneously deliver its unique portion of the data to each access module processor (AMP) controlled by that gateway;
- and
- instructing each AMP within each gateway to load a subset of that gateway's unique portion of the data to a table of the DBMS.
A system, comprising:
- one or more processors of a parallel database management system (DBMS) configured with fast loader, the fast loader executing on the one or ore processors;
- and
- wherein the fast loader configures clients so that each client acquires a unique portion of data from an external distributed file system (DFS) file and each client communicates its unique portion of data to a specific gateway of a database management system (DBMS), the fast loader also configures each gateway to deliver all of its unique portion of data to all access module processors on that gateway, and the fast loader configures each AMP to load a subset of its unique portion of data to a table of the DBMS in parallel with remaining AMPs.
Dependencies are inferred from the claim text.