System and method for cloud-to-cloud migration of data

A computer-implemented method for migrating data from a first object storage service to a second object storage service, the method comprising:

• receiving a migration request by a management module from a user;
• identifying migration parameters based on data metrics collected from the first object storage service and the second object storage service;
• determining a number of migration slots;
• deploying a plurality of engines based on the migration parameters;
• assigning a unique migration slot identifier to each engine of the plurality of engines;
• independently listing, by each engine, objects stored in the first object storage service;
• calculating, for each listed object, a hash value of an object key;
• determining whether the object belongs to the assigned migration slot based on a modulo operation using the hash value and the number of migration slots;
• providing tasks to the plurality of engines by the management module to migrate data from the first object storage service to the second object storage service;
• and
• writing data objects in the second object storage service, and
• wherein the plurality of engines migrate objects without requiring inter-engine communication during migration.

A computer-implement system for migrating object data from a source object storage to a destination object storage, the system comprising:

• a memory;
• and
• one or more processors to execute computer-readable instructions stored in the memory to perform a method comprising:
• receiving a migration request by a management module from a user;
• identifying migration parameters based on data metrics collected from the source object storage and the destination object storage;
• determining a number of migration slots;
• deploying a plurality of engines based on the migration parameters;
• assigning a unique migration slot identifier to each engine of the plurality of engines;
• receiving by a migration manager of the management module the tasks from tasks manger;
• and assigning the migration slot to each engine of the plurality of engines;
• receiving the migration status reports by a report processor from the plurality of engines and providing status reports to the task manager;
• receiving tasks requests by a task describer from the plurality of engines, identifying tasks parameters and providing the tasks descriptions to each engines;
• listing objects by a first lister from the first object storage service and listing objects by a second lister from the second object storage service;
• reading the objects by a migrator from the source object storage and writing the objects in the destination object storage;
• comparing by a comparator the objects from the source object storage and the objects in the destination object storage to verify whether the object to be migrated to the destination object storage and feeding the lists of objects to be migrated to the migrator, wherein the comparator compares the objects page-by-page;
• and
• writing data objects in the destination object storage,
• wherein migration objects are distributed pseudo-randomly across the plurality of engines without requiring communication between the plurality of engines during migration.

A computer-implemented system for storing data in multiple object storages simultaneously, the system comprising:

• a frontend abstraction configured to receive object-storage API requests from a client application, and translate the cloud specific requests to cloud-agnostic operations and translate responses associated with the cloud-agnostic operations into responses compatible with the object-storage API;
• a multi-cloud proxy configured to receive the cloud-agnostic operations and process the cloud-agnostic operations to the multiple object storages by generating corresponding cloud-specific operations for each of a plurality of object storages, transmitting the cloud-specific operations to the plurality of object storages, and aggregating responses received from the plurality of object storages;
• a continuation token combiner/parser configured to determine a unitary continuation token by combining individual continuation tokens received from at least two object storages, wherein each individual continuation token is determined based on an individual cloud identification, a cloud list page offset and a cloud continuation token, wherein the continuation token combiner/parser is further configured to parse the unitary continuation token to recover the continuation token associated with each object storage;
• a multipart-upload identifier combiner/parser configured to generate a combined multipart-upload identifier from a plurality of storage-specific multipart-upload identifiers received from the plurality of object storages and parse the combined multipart-upload identifier to recover the storage-specific multipart-upload identifier associated with each object storage;
• and and
• a version identifier combiner/parser configured to generate a combined version identifier from a plurality of storage-specific version identifiers received from the plurality of object storages and parse the combined version identifier to recover the storage-specific version identifier associated with each object storage,
• wherein the frontend abstraction, the multi-cloud proxy, the continuation token combiner/parser, the multipart-upload identifier combiner/parser, and the version identifier combiner/parser collectively provide a cloud-specific object-storage interface that enables a client application designed for a single object-storage service to simultaneously access and store data across the plurality of cloud-agnostic object storages.