Table level distributed database system for big data storage and query

A method of managing on-line user data in a table-level distributed database system, comprising:

• receiving, at a loading interface of the table-level distributed database system, a load request from a computational server specifying a new or updated data table to load into the table-level distributed database system, wherein the new or updated data table has been populated in a distribute file system, and wherein the load request includes loading parameters including a location in the distributed file system and a scheme of the new or updated data table;
• creating, by the loading interface, a plurality of metadata records to be stored in a metadata storage system,
• wherein the metadata records include the loading parameters including the location and scheme information for the new or updated data table
• wherein the metadata records specify a mapping between the new or updated data table and a particular one of a plurality of storage instances in the table-level database system that are to load such new or updated data table,
• wherein the metadata records are stored as multiple replicas in the metadata storage system;
• loading, at each particular one of the storage instances, the new or updated data table from the distributed file system in response to such particular one of the storage instances being mapped to the new or updated data table in at least one of the metadata records, wherein each storage instance includes:
• a local relational database management system (RDMS) instance, and
• an agent for loading or dropping mapped data tables, including the mapped new or updated data table, by periodically and asynchronously polling the metadata storage system to obtain loading parameters for the mapped data tables such that the mapped data tables are loaded in parallel;
• and
• publishing, by each particular storage instance, the load status information to the metadata records in the metadata storage system, the load status information indicating that loading of the new or updated data table is complete after such new or updated data table's loading is complete so that such new or updated data table can be queried by a query client.

The method of 13, further comprising:

• removing the new or updated data table by any one of the storage instances if such storage instance determines that the new or updated data table is no longer mapped to such storage instance in the metadata storage system;
• unpublishing, at the metadata storage system, load status information for any one of the storage instances and its mapped new or updated data table when it's unavailable at such storage instance;
• and
• republishing any unpublished load status information for the new or updated data table and any one of the storage instances when such storage instance's new or updated data table becomes available again after being unavailable and if such new or updated data table is determined to be valid.