Method and system for efficient content transfer to a server

A method comprising:

• receiving, from a client application, a request to store first content to a distributed data store;
• determining a size of the first content;
• determining a chunk size;
• determining a first plurality of content chunks to create based on the size of the first content and the chunk size and creating the first plurality of content chunks from the first content;
• assigning each content chunk in the first plurality of content chunks a chunk id that is unique to that content chunk in the first plurality of content chunks;
• communicating with a server to store the first content as an object in the distributed data store, the object comprising object metadata and associated content, the communicating with the server to store the first content comprising:
• establishing a first plurality of content transfer sessions with the server to transfer the first plurality of content chunks to the server for storage in the distributed data store as the associated content of the object, including establishing at least two parallel content transfer sessions to transfer at least two of the content chunks from the first plurality of content chunks to the server in parallel;
• and
• transferring each content chunk from the first plurality of content chunks and chunk metadata for that content chunk to the server using a corresponding content transfer session from the first plurality of content transfer sessions, the chunk metadata including the chunk id for that chunk and an indication of the object to which the chunk is to be stored;
• receiving a request to access the object;
• establishing a metadata transfer session with the server to transfer the object metadata from the server and receiving the object metadata from the server, the object metadata including a first set of hash values, the first set of hash values including a hash value generated from each content chunk in the first plurality of content chunks, each hash value in the first set of hash values having an associated hash value index value;
• receiving a request to store second content to the object;
• creating, from the second content, a first content chunk of the chunk size, assigning the first content chunk a first index value and generating a current hash value from the first content chunk;
• selecting a first hash value from the first set of hash values based on the associated hash value index value associated with the first hash value matching the first index value;
• comparing the first hash value to the current hash value generated from the first content chunk;
• based on a determination that the first hash value and the current hash value do not match, transferring the first content chunk to the server to store the first content chunk in the object;
• and
• based on a determination that the first hash value and the current hash value match, skipping the first content chunk.

A system for storing data comprising:

• a distributed data store;
• a client computer comprising a client application;
• a content management computer system coupled to the distributed data store, the content management computer system comprising a server and a client interface, the client interface comprising instructions for:
• receiving a request from the client application to store first content to the distributed data store;
• determining a size of the first content;
• determining a chunk size;
• determining a first plurality of content chunks to create based on the size of the first content and the chunk size and creating the first plurality of content chunks from the first content;
• assigning each content chunk in the first plurality of content chunks a chunk id that is unique to that content chunk in the first plurality of content chunks;
• communicating with the server to store the first content as an object in the distributed data store, the object comprising object metadata and associated content, the communicating with the server to store the first content comprising:
• establishing a first plurality of content transfer sessions with the server to transfer the first plurality of content chunks to the server for storage in the distributed data store as the associated content of the object, including establishing at least two parallel content transfer sessions to transfer at least two of the content chunks from the first plurality of content chunks to the server in parallel;
• and
• transferring each content chunk from the first plurality of content chunks and chunk metadata for that content chunk to the server using a corresponding content transfer session from the first plurality of content transfer sessions, the chunk metadata including the chunk id for that chunk and an indication of the object to which the chunk is to be stored;
• receiving a request from the client application to access the object;
• establishing a metadata transfer session with the server to transfer the object metadata from the server and receiving the object metadata from the server, the object metadata including a first set of hash values;
• receiving, from the client application, a request to store second content to the object, the second content being a modified version of the first content;
• creating, from the second content, a first content chunk of the chunk size, assigning the first content chunk a first index value and generating a current hash value from the first content chunk;
• selecting a first hash value from the first set of hash values based on an associated hash value index value associated with the first hash value matching the first index value;
• comparing the first hash value to generated from the first content chunk;
• based on a determination that the first hash value and the current hash value do not match, transferring the first content chunk to the server to store the first content chunk in the object;
• and
• based on a determination that the first hash value and the current hash value match, skipping the first content chunk;
• and
• the server comprising instructions for:
• generating the first set of hash values from the first plurality of content chunks, the first set of hash values including a hash value generated from each content chunk in the first plurality of content chunks, the first set of hash values associated with a set of associated hash value index values;
• and
• storing the first set of hash values in the object metadata.

A computer program product comprising a non-transitory computer readable medium having stored therein a set of computer-readable instructions, including instructions for:

• receiving a request to store first content to a distributed data store;
• determining a size of the first content;
• determining a chunk size;
• determining a first plurality of content chunks to create based on the size of the first content and the chunk size and creating the first plurality of content chunks from the first content;
• assigning each content chunk in the first plurality of content chunks a chunk id that is unique to that content chunk in the first plurality of content chunks;
• communicating with a server to store the first content as an object in the distributed data store, the object comprising object metadata and associated content, the communicating with the server to store the first content comprising:
• establishing a first plurality of content transfer sessions with the server to transfer the first plurality of content chunks to the server for storage in the distributed data store as the associated content of the object, including establishing at least two parallel content transfer sessions to transfer at least two of the content chunks from the first plurality of content chunks to the server in parallel;
• and
• transferring each content chunk from the first plurality of content chunks and chunk metadata for that content chunk to the server using a corresponding content transfer session from the first plurality of content transfer sessions, the chunk metadata including the chunk id for that chunk and an indication of the object to which the chunk is to be stored;
• receiving a request to access the object;
• establishing a metadata transfer session with the server to transfer the object metadata from the server and receiving the object metadata from the server, the object metadata including a first set of hash values, the first set of hash values including a hash value for each content chunk in the first plurality of content chunks;
• receiving a request to store second content to the object, the second content being a modified version of the first content;
• creating a second plurality of content chunks from the second content;
• generating a second set of hash values, the second set of hash values including a hash value for each content chunk in the second plurality of content chunks;
• determining which content chunks in the second plurality of content chunks are unmodified content chunks based on comparing the first set of hash values to the second set of hash values;
• and
• communicating with the server to store the second content to the object, the communicating with the server to store the second content to the object comprising establishing at least one session to transfer the content chunks from the second plurality of content chunks, other than the unmodified content chunks, to the server.

A method comprising:

• receiving, from a client application, a request to store first content to a distributed data store;
• determining a size of the first content;
• determining a chunk size;
• determining a first plurality of content chunks to create based on the size of the first content and the chunk size and creating the first plurality of content chunks from the first content;
• assigning each content chunk in the first plurality of content chunks a chunk id that is unique to that content chunk in the first plurality of content chunks;
• communicating with a server to store the first content as an object in the distributed data store, the object comprising object metadata and associated content, the communicating with the server to store the first content comprising:
• establishing a first plurality of content transfer sessions with the server to transfer the first plurality of content chunks to the server for storage in the distributed data store as the associated content of the object, including establishing at least two parallel content transfer sessions to transfer at least two of the content chunks from the first plurality of content chunks to the server in parallel;
• and
• transferring each content chunk from the first plurality of content chunks and chunk metadata for that content chunk to the server using a corresponding content transfer session from the first plurality of content transfer sessions, the chunk metadata including the chunk id for that chunk and an indication of the object to which the chunk is to be stored;
• receiving a request to access the object;
• establishing a metadata transfer session with the server to transfer the object metadata from the server and receiving the object metadata from the server, the object metadata including a first set of hash values, the first set of hash values including a hash value for each content chunk in the first plurality of content chunks;
• receiving a request to store second content to the object, the second content being a modified version of the first content;
• creating a second plurality of content chunks from the second content;
• generating a second set of hash values, the second set of hash values including a hash value for each content chunk in the second plurality of content chunks;
• determining which content chunks in the second plurality of content chunks are unmodified content chunks based on comparing the first set of hash values to the second set of hash values;
• and
• communicating with the server to store the second content to the object, the communicating with the server to store the second content to the object comprising establishing at least one session to transfer the content chunks from the second plurality of content chunks, other than the unmodified content chunks, to the server.

A system for storing data comprising:

• a distributed data store;
• a client computer comprising a client application;
• a content management computer system coupled to the distributed data store, the content management computer system comprising a server and a client interface, the client interface comprising instructions for:
• receiving a request from the client application to store first content to the distributed data store;
• determining a size of the first content;
• determining a chunk size;
• determining a first plurality of content chunks to create based on the size of the first content and the chunk size and creating the first plurality of content chunks from the first content;
• assigning each content chunk in the first plurality of content chunks a chunk id that is unique to that content chunk in the first plurality of content chunks;
• communicating with the server to store the first content as an object in the distributed data store, the object comprising object metadata and associated content, the communicating with the server to store the first content comprising:
• establishing a first plurality of content transfer sessions with the server to transfer the first plurality of content chunks to the server for storage in the distributed data store as the associated content of the object, including establishing at least two parallel content transfer sessions to transfer at least two of the content chunks from the first plurality of content chunks to the server in parallel;
• and
• transferring each content chunk from the first plurality of content chunks and chunk metadata for that content chunk to the server using a corresponding content transfer session from the first plurality of content transfer sessions, the chunk metadata including the chunk id for that chunk and an indication of the object to which the chunk is to be stored;
• receiving a request to access the object;
• establishing a metadata transfer session with the server to transfer the object metadata from the server and receiving the object metadata from the server, the object metadata including a first set of hash values, the first set of hash values including a hash value for each content chunk in the first plurality of content chunks;
• receiving a request to store second content to the object, the second content being a modified version of the first content;
• creating a second plurality of content chunks from the second content;
• generating a second set of hash values, the second set of hash values including a hash value for each content chunk in the second plurality of content chunks;
• determining which content chunks in the second plurality of content chunks are unmodified content chunks based on comparing the first set of hash values to the second set of hash values;
• and
• communicating with the server to store the second content to the object, the communicating with the server to store the second content to the object comprising establishing at least one session to transfer the content chunks from the second plurality of content chunks, other than the unmodified content chunks, to the server;
• and
• the server comprising instructions for:
• generating the first set of hash values from the first plurality of content chunks, the first set of hash values including a hash value generated from each content chunk in the first plurality of content chunks, the first set of hash values associated with a set of associated hash value index values;
• and
• storing the first set of hash values in the object metadata.