Deduplication and garbage collection across logical databases

A method executed by a first node of a plurality of nodes in communication with a storage system that stores a plurality of external resources, wherein each node of the plurality of nodes maintains a different respective logical database portion that references a respective portion of the external resources stored by the storage system, each respective logical database portion further being associated with a different respective set of one or more users, the method comprising:

• receiving, by the first node, from a computing device associated with a first user of a first set of one or more users associated with a first logical database portion maintained at the first node, a request to add a first data object;
• searching, by the first node, a first reference data structure included in the first logical database portion maintained at the first node, the first reference data structure including, for each external resource referenced by the first logical database portion on the first node, a reference count of a number of references in the respective first logical database portion to each individual external resources referenced by the first set of one or more users, wherein the searching comprises searching for an entry in the first reference data structure referring to a particular external resource having content that matches content of the first data object;
• in response to determining that a reference to the particular external resource does not exist in the first reference data structure included in the first logical database portion on the first node, sending a query to at least one other node of the plurality of nodes to determine whether a reference to the particular external resource is maintained in a second reference data structure included in a second logical database portion maintained on the at least one other node;
• and
• in response to receiving an indication from a second node of the plurality of nodes that a reference to the particular external resource is maintained in the second reference data structure included in the second logical database portion maintained on the second node, creating a new entry for the first data object in the first reference data structure included in the first logical database portion on the first node without saving the first data object to the storage system.

A system enabling horizontal scaling of data across a plurality of logical database portions, the system comprising:

• a storage system storing a plurality of files as external resources;
• and
• a plurality of nodes including the plurality of logical database portions, such that each node includes a different respective logical database portion, each logical database portion including metadata that references a subset of the plurality of external resources, each respective logical database portion further being associated with a different respective set of one or more users, wherein:
• each node is programmed to maintain, for each external resource referenced by the respective logical database portion on that node by the respective set of one or more users associated with the respective logical database portion, as part of the respective logical database portion, a respective local global table that includes a reference count of a number of references in the respective logical database portion to each external resource, a resource identifier (ID) that is based at least in part on content of a corresponding external resource, and time information that is associated with the reference count;
• and
• in response to addition of a reference to a particular external resource in a first one of the logical database portions, by a user of a first set of one or more users associated with the first logical database portion, a first node including the first logical database portion is programmed to increment the reference count for the particular external resource in a first local global table included in the first logical database portion maintained by the first node, and update the time information associated with the reference count for the particular external resource based at least in part on a time associated with the addition of the reference to the particular external resource.