Back to results
US 10,691,558 B1Granted
Fault tolerant data export using snapshots
AssigneeAmazon Technologies, Inc.
InventorsNitin Kesarwani, Sheikh Naveed Zafar
Life of this patent
Estimating the term…
Abstract
Fault tolerant data export is provided through the checkpointing of various export tasks. Data entries, such as event log records, to be exported are assigned to a set of discrete export tasks. A snapshot of the data can be taken to indicate the state of the data to be exported. The tasks can be performed using appropriate resources, and for each task that completes successfully an entry can be made in a checkpoint data file or a virtual cursor can be updated for the snapshot indicating completion of that task. In the event of a failure of one or more tasks, a workflow manager or other such component can determine which tasks failed using the checkpointing data and retry only those specific tasks, without having to retry the entire export.
Claims17
A computer-implemented method, comprising:
- receiving a request to export customer log data generated over a specified period of time;
- determining the customer log data corresponding to the request, the customer log data being stored to a plurality of storage locations across a resource provider environment;
- generating a snapshot indicating a state of the customer log data at a time of the request, wherein the snapshot includes at least one status identifier to identify a partial completion of the request;
- assigning portions of the customer log data to a set of tasks, each task of the set of tasks responsible for writing a respective portion of the customer log data to a specified repository, wherein each respective portion is an independent and discrete portion of the customer log data;
- causing the tasks to be executed by one or more computing resources in the resource provider environment;
- updating the at least one status identifier with respect to the snapshot in response to one or more completed tasks, of the set of tasks, completing successfully in writing the respective portion of the customer log data to the specified repository;
- determining that at least one task, of the set of tasks, failed to complete successfully;
- determining, based at least in part upon the at least one status identifier and the snapshot, the at least one task that failed to complete successfully;
- causing the at least one task to be executed at least a second time by the one or more computing resources without re-executing the completed task of the set of tasks;
- causing the at least one task to be re-executed until the at least one task completes successfully or a maximum number of retries is reached for the request;
- and
- providing less than the requested log data if the at least one task fails to complete, wherein the provided log data includes log data retrieved by the one or more completed tasks.
9 more elements
A computer-implemented method, comprising:
- determining a set of data to be exported from at least one data repository;
- obtaining a snapshot indicating a state of the set of data to be exported, wherein the snapshot includes a status identifier to identify partial exportation of the set of data to be exported;
- assigning portions of the set of data to a set of tasks, each task causing a respective portion of the set of data to be exported, wherein each respective portion is an independent and discrete portion of the set of data;
- causing at least a subset of the set of tasks to be executed;
- updating the status identifier for the set of data in response to one or more completed tasks, of the set of tasks, completing successfully;
- determining that at least a failed task, of the set of tasks, failed to complete successfully;
- identifying, based at least in part upon the status identifier and the snapshot, the failed task and one or more successfully completed tasks;
- causing the failed task to be executed at least a second time without re-executing the one or more successfully completed tasks;
- causing the failed task to be executed until the failed task completes successfully or a maximum number of retries is reached for the failed task;
- and
- providing less than the requested log data if the at least one task fails to complete, wherein the provided log data includes log data retrieved by the one or more completed tasks.
8 more elements
A system, comprising:
- at least one processor;
- and
- memory including instructions that, when executed by the at least one processor, cause the system to:
- determine a set of data to be exported from at least one data repository;
- obtain a snapshot indicating a state of the set of data to be exported, wherein the snapshot includes a status identifier to identify partial exportation of the set of data to be exported;
- assign portions of the set of data to a set of tasks, each task causing a respective portion of the set of data to be exported to a specified location, wherein each respective portion is an independent and discrete portion of the customer log data;
- cause at least a subset of the set of tasks to be executed;
- update the status identifier for the set of data in response to one or more completed tasks, of the set of tasks, completing successfully;
- determine that at least a failed task, of the set of tasks, failed to complete successfully;
- identify, based at least in part upon the status identifier and the shapshot, the failed task and one or more successfully completed tasks;
- cause the failed task to be executed at least a second time without re-executing the one or more successfully completed tasks;
- cause the failed task to be executed until the failed task completes successfully or a maximum number of retries is reached for the failed task;
- and
- provide less than the requested log data if the at least one task fails to complete, wherein the provided log data includes log data retrieved by the one or more completed tasks.
11 more elements
Dependencies are inferred from the claim text.