# US 2015/0169407 A1 · Method and system for error detection and correction in append-only datastores

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0169407 A1
- Document ID: US20150169407A1
- Status: Published application
- Kind: A1
- Application number: 14/569,434
- Filed: 2014-12-12
- Published: 2015-06-18
- Earliest claimed: 2013-12-16
- Assignees: Deep Information Sciences, Inc.
- Applicants: Deep Information Sciences, Inc.
- Inventors: Thomas HAZEL; Gerry BUTEAU; David NOBLET; Jason JEFFORDS
- Web page: https://patentgrep.com/patent/US20150169407A1

## CPC

- G06F 11/1004: {to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method, system, and computer program product for error detection and correction in append-only data-stores. A user or agent input is received. Error detection and correction involving at least one datastore is begun based on the user or agent input. An error detection and correction state is created and error detection and correction is performed. A file sealing indication and/or a clean shutdown indication is interpreted, the error-detection and correction is ended, and the state of the error-detection and correction is written to memory in an append-only manner.

## Claims

### Claim 1

A computer assisted method for error detection and correction in append-only data-stores, the method including:
receiving a user or agent input;
beginning error detection and correction involving at least one datastore based on the user or agent input;
creating an error detection and correction state;
performing error detection and correction;
interpreting at least one of a file sealing indication and a clean shutdown indication;
ending the error-detection and correction; and
writing the state of the error-detection and correction to memory in an append-only manner.

### Claim 2 (depends on 1)

The method of claim 1 wherein the state comprises at least one selected from a group consisting of error-detection and correction indication and erasure code.

### Claim 3 (depends on 1)

The method of claim 1, wherein the state comprises error-detection and correction indication and erasure code.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
updating the error and correction state; and
maintaining the error and correction state.

### Claim 5 (depends on 1)

The method of claim 1, wherein the error-detection and correction state is represented in an encoded state by at least one selected from a group consisting of an append-only transaction log, key, value and schema files.

### Claim 6 (depends on 5)

The method of claim 5, wherein the memory is disk memory.

### Claim 7 (depends on 1)

The method of claim 1, wherein beginning the error-detection and correction further includes accessing at least one file within the datastore to perform error detection and correction within the file.

### Claim 8 (depends on 7)

The method of claim 7, wherein error detection and correction within the file includes file header error detection and correction.

### Claim 9 (depends on 8)

The method of claim 8, wherein header error detection further comprises:
header length decoding and validation;
optional header CRC and CRC validation; and
header content validation.

### Claim 10 (depends on 8)

The method of claim 8, wherein header error correction further comprises:
header length detection;
header regeneration; and
file regeneration.

### Claim 11 (depends on 7)

The method of claim 7, wherein error detection and correction within a file includes record validation.

### Claim 12 (depends on 11)

The method of claim 11, wherein the records are of a fixed size.

### Claim 13 (depends on 11)

The method of claim 11, wherein the records are variable sized.

### Claim 14 (depends on 11)

The method of claim 11, wherein record validation includes data consistency and CRC checks.

### Claim 15 (depends on 7)

The method of claim 7, wherein error detection and correction within a file includes segment validation.

### Claim 16 (depends on 15)

The method of claim 15, wherein the segments are variable sized.

### Claim 17 (depends on 15)

The method of claim 15, wherein segments are compressed.

### Claim 18 (depends on 15)

The method of claim 15, wherein segment validation includes at least one selected from a group consisting of decompression, data consistency and CRC checks.

### Claim 19 (depends on 7)

The method of claim 7, wherein error detection and correction within a file includes frame validation.

### Claim 20 (depends on 19)

The method of claim 19, wherein the frames are variable sized.

### Claim 21 (depends on 19)

The method of claim 19, wherein frames are compressed.

### Claim 22 (depends on 19)

The method of claim 19, wherein frame validation includes decompression, data consistency and CRC checks.

### Claim 23 (depends on 7)

The method of claim 7, wherein error detection may be performed up to T Trials.

### Claim 24 (depends on 23)

The method of claim 23, wherein error detection is performed on frames.

### Claim 25 (depends on 23)

The method of claim 23, where error detection is performed on segments.

### Claim 26 (depends on 5)

The method of claim 5, wherein the append-only transaction log, key, value and schema files encode state represent file sealing and clean shutdown.

### Claim 27 (depends on 26)

The method of claim 26, further comprising:
appending clean shutdown codes to files to indicate a clean shutdown; and
appending file sealed codes to files to indicate file sealing.

### Claim 28 (depends on 26)

The method of claim 26, wherein file sealing and clean shutdown codes include an additional state used during startup and error recovery.

### Claim 29 (depends on 28)

The method of claim 28, further comprising:
including metadata comprising at least one selected from a group consisting of cardinality and statistical aggregations in index files; and
including at least one selected from a group consisting of indexing and summary indexing in index files

### Claim 30 (depends on 7)

The method of claim 7, wherein error detection and correction within a file further comprises interpreting at least one selected from a group consisting of a file sealing indication and clean shutdown indication.

### Claim 31 (depends on 30)

The method of claim 30, further comprising:
using clean shutdown and file sealing codes during startup; and
using an absence of the clean shutdown indication and the file sealing indication in order to identify a potential error.

### Claim 32 (depends on 1)

The method of claim 1, wherein error correction includes record erasure.

### Claim 33 (depends on 32)

The method of claim 32, wherein record erasure is achieved through the use of append-only erasure codes.

### Claim 34 (depends on 32)

The method of claim 32, wherein record erasure is achieved through write cursor adjustment including cursor rewind.

### Claim 35 (depends on 1)

The method of claim 1, further comprising performing an overall error detection and correction, wherein an overall error detection and correction includes correcting related database, datastore, transaction log, state log, index and schema files.

### Claim 36 (depends on 1)

The method of claim 1, wherein a shutdown request causes a clean shutdown indication to be appended to all files modified since a last shutdown.

### Claim 37 (depends on 1)

The method of claim 1, wherein a seal file request causes a file sealed indication to be appended to a file being sealed.

### Claim 38 (depends on 1)

The method of claim 1, wherein a start up request initiates a fast error detection process that checks each database, each datastore, and all contained files for at least one of a sealed status and a clean shutdown status.

### Claim 39

An automated system for error detection and correction in append-only data-stores, the system comprising:
means for receiving a user or agent input;
means for beginning error detection and involving at least one datastore based on the user or agent input;
means for updating and maintaining an error detection state;
means for correcting detected errors; and
means for writing the error detection and correction state to memory in an append-only manner.

### Claim 40 (depends on 39)

The automated system of claim 39, wherein the error detection and correction state comprises append-only transaction logs, key, value and schema files.

### Claim 41

A computer program product comprising a computer readable medium having control logic stored therein for causing a computer to perform error detection and correction in append-only data-stores, the control logic comprising code for:
receiving a user or agent input;
beginning error detection and correction involving at least one datastore based on the user or agent input;
updating and maintaining an error detection and correction state;
ending the error detection and correction; and
writing the state of the error detection and correction to memory in an append-only manner.

### Claim 42 (depends on 41)

The computer program product of claim 41, wherein the error detection and correction state comprises append-only transaction logs, key, value and schema files.

### Claim 43

An automated system of error detection and correction in append-only data-stores, the system comprising:
at least one processor;
a user interface functioning via the at least one processor, wherein the user interface is configured to receive a user input; and
a repository accessible by the at least one processor; wherein the at least one processor is configured to:
begin a transaction involving at least one datastore based on the user input;
update and maintain an error detection and correction state;
end the error detection and correction; and
write the state of the error detection and correction to memory in an append-only manner.

### Claim 44 (depends on 43)

The automated system of claim 43, wherein the error detection and correction state comprises append-only transaction log, key, value and schema files.
