Handling semi-structured and unstructured data in a sharded database environment

A method comprising:

• receiving, at a catalog server from a requesting entity, a cross-shard query;
• in response to receiving the cross-shard query, the catalog server sending remote commands to a plurality of shard servers;
• in response to sending the remote commands, receiving at the catalog server a lob locator from each shard server of the plurality of shard servers;
• wherein the plurality of shard servers includes a particular shard server that provided a particular lob locator to the catalog server in response to the remote commands;
• the catalog server encoding the particular lob locator with a shard identifier to produce an encoded lob locator that identifies the particular shard server;
• responding to the cross-shard query by providing the encoded lob locator to the requesting entity;
• receiving, at the catalog server from the requesting entity, a fetch request associated with the encoded lob locator;
• based on the shard identifier in the encoded lob locator, determining at the catalog server that the fetch request is to be pushed to the particular shard server;
• and
• the catalog server pushing the fetch request to the particular shard server.

A method comprising:

• receiving, at a catalog server, a database command that references items in a relational table that has been sharded into a plurality of shards;
• wherein each shard of the plurality of shards is assigned to a corresponding shard server of a plurality of shard servers;
• determining, at the catalog server, that execution of the database command requires a particular operation;
• wherein the particular operation is one of:
• an operation specified by a semi-structured SQL operator in the database command;
• an operation that creates an index from data extracted from semi-structured items in the relational table;
• an operation that specifies a relational view of semi-structured data extracted from semi-structured items in the relational table;
• an operation that specifies generation of a data guide based on semi-structured items in the relational table;
• an operation that specifies a semi-structured data view of relational data stored in the relational table;
• sending a remote command to each shard server of the plurality of shard servers to cause each shard server to perform the particular operation on items in its respective shard.

A method comprising:

• receiving, at a catalog server, a database command that references items in a relational table that has been sharded into a plurality of shards;
• wherein the sharded table stores semi-structured items;
• wherein the semi-structured items are self-contained relative to a particular level of granularity;
• wherein each shard of the plurality of shards is assigned to a corresponding shard server of a plurality of shard servers;
• determining, at the catalog server, that execution of the database command requires a particular operation;
• sending a remote command to each shard server of the plurality of shard servers to cause each shard server to perform the particular operation on items in its respective shard;
• and
• migrating the semi-structured items among the plurality of shards, during a re-sharding operation, at the same particular level of granularity as the level of granularity at which the semi-structured items are self-contained.

One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:

• receiving, at a catalog server from a requesting entity, a request to perform a cross-shard query involving unstructured or semi-structured data;
• the catalog server obtaining the unstructured or semi-structured data from one or more shard servers;
• after receiving the unstructured or semi-structured data from the one or more shard servers, the catalog server creating one or more temporary lobs, and responding to the request by providing the requesting entity one or more lob locators that correspond to the one or more temporary lobs;
• and
• in response to fetch operations initiated by the requesting entity based on any of the one or more lob locators, the catalog server accessing the one or more temporary lobs stored within the catalog server, without involving the one or more shard servers.

The one or more non-transitory computer-readable media of claim 27 wherein the one or more shard servers provide the unstructured or semi-structured data to the catalog server by:

• creating temporary lobs;
• and
• providing the catalog server lob locators to the temporary lobs;
• wherein the catalog server obtains all items needed from the temporary lobs created by the one or more shard servers, and releases the lob locators to the temporary lobs created by the one or more shard servers, before performing fetch operations initiated by the requesting entity based on any of the one or more lob locators.

One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:

• receiving, at a catalog server from a requesting entity, a cross-shard query;
• in response to receiving the cross-shard query, the catalog server sending remote commands to a plurality of shard servers;
• in response to sending the remote commands, receiving at the catalog server a lob locator from each shard server of the plurality of shard servers;
• wherein the plurality of shard servers includes a particular shard server that provided a particular lob locator to the catalog server in response to the remote commands;
• the catalog server encoding the particular lob locator with a shard identifier to produce and encoded lob locator that identifies the particular shard server;
• responding to the cross-shard query by providing the encoded lob locator to the requesting entity;
• receiving, at the catalog server from the requesting entity, a fetch request associated with the encoded lob locator;
• based on the shard identifier in the encoded lob locator, determining at the catalog server that the fetch request is to be pushed to the particular shard server;
• and
• the catalog server pushing the fetch request to the particular shard server.

One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:

• receiving, at a catalog server, a database command that references items in a relational table that has been sharded into a plurality of shards;
• wherein each shard of the plurality of shards is assigned to a corresponding shard server of a plurality of shard servers;
• determining, at the catalog server, that execution of the database command requires a particular operation;
• wherein the particular operation is one of:
• an operation specified by a semi-structured SQL operator in the database command;
• an operation that creates an index from data extracted from semi-structured items in the relational table;
• an operation that specifies a relational view of semi-structured data extracted from semi-structured items in the relational table;
• an operation that specifies generation of a data guide based on semi-structured items in the relational table;
• an operation that specifies a semi-structured data view of relational data stored in the relational table;
• sending a remote command to each shard server of the plurality of shard servers to cause each shard server to perform the particular operation on items in the shard, of the plurality of shards, that are assigned to the shard server.

The one or more non-transitory computer-readable media of claim 30 wherein the instructions comprise instructions for:

• receiving, at the catalog server, a second database command that references items in the relational table;
• based on (a) a predicate of the second database command that references an attribute of semi-structured data, and (b) information obtained from data guides associated with each shard of the plurality of shards, determining that only a subset of shards of the plurality of shards could possibly have items that satisfy the predicate;
• and
• the catalog server responding to the second database command by sending remote commands to only the shard servers, of the plurality of shard servers, that correspond to the subset of shards.