Scale out data storage and query filtering using data pools

A method, implemented at a computer system that includes one or more processors, for performing a distributed query across a data pool, the method comprising:

• receiving a first database query at a master node or a compute pool within a database system, the first database query targeting an external data source;
• based on receiving the first database query, and based on the first database query being a read query:
• identifying a data pool within the database system, in which, (i) the data pool comprises a plurality of data nodes, each data node including a relational engine and relational storage;
• and (ii) each data node caches a different partition of data from the external data source in its relational storage;
• processing the first database query across the plurality of data nodes, including requesting that each data node perform a filter operation against its cached partition of the external data source stored in its relational storage, and return any data from the partition that matches the filter operation;
• receiving a second database query at the master node or the compute pool within the database system;
• based on receiving the second database query, and based on the second database query being a write query, routing the second database query to the external data source, which modifies at least one record in the external data source;
• and
• subsequent to routing the second database query to the external data source, ingesting the modified at least one record from the external data source into the data pool.

A computer program product comprising hardware storage devices having stored thereon computer-executable instructions that, when executed at one or more processors, cause a computer system to perform the following:

• receive a first database query at a master node or a compute pool within a database system, the first database query targeting an external data source;
• based on receiving the first database query, and based on the first database query being a read query:
• identify a data pool within the database system, in which, (i) the data pool comprises a plurality of data nodes, each data node including a relational engine and relational storage;
• and (ii) each data node caches a different partition of data from the external data source in its relational storage;
• and
• process the database query across the plurality of data nodes, including requesting that each data node perform a filter operation against its cached partition of the external data source stored in its relational storage, and return any data from the partition that matches the filter operation;
• receive a second database query at the master node or the compute pool within the database system;
• based on receiving the second database query, and based on the second database query being a write query, route the second database query to the external data source, which modifies at least one record in the external data source;
• and
• subsequent to routing the second database query to the external data source, ingest the modified at least one record from the external data source into the data pool.