US 8,276,022 B2 · Efficient failure detection for long running data transfer jobs

Abstract

Disclosed are methods and apparatus for error handling within jobs that utilize a plurality of tasks for data transfer of individual data records to a storage destination. For each task, one or more failed records may be logged to a file for later insertion. If a high percentage of a task's output (e.g., writes to another data storage system) is determined to be failing, the task short-circuits itself. Each task is also configured to perform checkpoint logging as the task completes work. If the entire job later short-circuits and is to be restarted, the restarted job only repeats a minimal amount of previously completed work for the tasks which have not already completed their data insertions. Together, these techniques can ensure that in the face of periodic failures, the job completes long-running job in a minimal time with minimal effects.

Claims18

Dependent claims

A computer implemented method of handling errors during a data transfer, comprising:

(a) for a first task that is configured to transfer a plurality of data records from a source to a destination storage system and when a specific record of such first task fails to be transferred to the destination storage system, causing the first task to retry transferring of the specific record to the destination storage system so that such retry is only performed a predefined number of times;
(b) when the first task has been caused to retry transferring of a specific record of the first task more than the predefined number of times, storing the specific record in an error log for a later transfer attempt;
(c) periodically determining whether the first task's failure rate is above a predetermined threshold;
and
(d) if the first task's failure rate is determined to be above the predetermined threshold, short-circuiting the first task by inhibiting further transfer by the first task and identifying the first task as being short-circuited.

2 more elements

An apparatus comprising at least a processor and a memory, wherein the processor and/or memory are configured to perform the following operations:

(a) for a first task that is configured to transfer a plurality of data records from a source to a destination storage system and when a specific record of such first task fails to be transferred to the destination storage system, causing the first task to retry transferring of the specific record to the destination storage system so that such retry is only performed a predefined number of times;
(b) when the first task has been caused to retry transferring of a specific record of the first task more than the predefined number of times, storing the specific record in an error log for a later transfer attempt;
(c) periodically determining whether the first task's failure rate is above a predetermined threshold;
and
(d) if the first task's failure rate is determined to be above the predetermined threshold, short-circuiting the first task by inhibiting further transfer by the first task and identifying the first task as being short-circuited.

2 more elements

A computer program product for partitioning a native table in a database, comprising at least one computer-readable non-transitory medium having computer instructions stored therein which are operable to cause a computer device to perform the following operations:

(a) for a first task that is configured to transfer a plurality of data records from a source to a destination storage system and when a specific record of such first task fails to be transferred to the destination storage system, causing the first task to retry transferring of the specific record to the destination storage system so that such retry is only performed a predefined number of times;
(b) when the first task has been caused to retry transferring of a specific record of the first task more than the predefined number of times, storing the specific record in an error log for a later transfer attempt;
(c) periodically determining whether the first task's failure rate is above a predetermined threshold;
and
(d) if the first task's failure rate is determined to be above the predetermined threshold, short-circuiting the first task by inhibiting further transfer by the first task and identifying the first task as being short-circuited.

2 more elements

Dependencies are inferred from the claim text.

Efficient failure detection for long running data transfer jobs

Life of this patent

Efficient failure detection for long running data transfer jobs

Life of this patent

Abstract

Claims18