Background format optimization for enhanced SQL-like queries in Hadoop

A method for performing queries on stored data in a Hadoop™ distributed computing cluster, the method comprising:

• forming, based on a plurality of data nodes, a peer-to-peer network for the queries received from a client, a respective data node of the plurality of data nodes functioning as a peer in the peer-to-peer network and being capable of interacting with components of the Hadoop™ cluster;
• and
• in the respective data node, operating an instance of a query engine that is configured to:
• parse a query from a client;
• selectively creates query fragments based on an availability of converted data at the respective data node, the converted data corresponding to data associated with the query, wherein the converted data is the data associated with the query converted from an original format into a target format that is specified by a schema, and wherein the query is processed into said query fragments by whichever data node that receives the query;
• distribute the query fragments among the plurality of data nodes;
• execute the query fragments on whichever local data that corresponds to a format for which the query fragments are created, based on the schema;
• obtain intermediate results from other data nodes that receive the query fragments;
• and
• aggregate the intermediate results for the client.

A non-transitory machine-readable storage medium having stored thereon instructions which, when executed by one or more processors, cause the processors to perform a method comprising:

• forming, based on a plurality of data nodes, a peer-to-peer network for the queries received from a client, a respective data node of the plurality of data nodes functioning as a peer in the peer-to-peer network and being capable of interacting with components of a Hadoop™ cluster;
• and
• in the respective data node, operating an instance of a query engine that is configured to:
• parse a query from a client;
• selectively creates query fragments based on an availability of converted data at the respective data node, the converted data corresponding to data associated with the query, wherein the converted data is the data associated with the query converted from an original format into a target format that is specified by a schema, and wherein the query is processed into said query fragments by whichever data node that receives the query;
• distribute the query fragments among the plurality of data nodes;
• execute the query fragments on whichever local data that corresponds to a format for which the query fragments are created, based on the schema;
• obtain intermediate results from other data nodes that receive the query fragments;
• and
• aggregate the intermediate results for the client.