# US 2013/0159364 A1 · Parallel log structured file system collective buffering to achieve a compact representation of scientific and/or dimensional data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0159364 A1
- Document ID: US20130159364A1
- Status: Published application
- Kind: A1
- Application number: 13/722,946
- Filed: 2012-12-20
- Published: 2013-06-20
- Earliest claimed: 2011-12-20
- Assignees: UT-Battelle, LLC - Oak Ridge National Laboratory; Los Alamos National Security, LLC
- Inventors: Gary A. Grider; Stephen W. Poole
- Web page: https://patentgrep.com/patent/US20130159364A1

## CPC

- G06F 16/1865: {Transactional file systems}
- G06F 16/1858: {Parallel file systems, i.e. file systems supporting multiple processors}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Collective buffering and data pattern solutions are provided for storage, retrieval, and/or analysis of data in a collective parallel processing environment. For example, a method can be provided for data storage in a collective parallel processing environment. The method comprises receiving data to be written for a plurality of collective processes within a collective parallel processing environment, extracting a data pattern for the data to be written for the plurality of collective processes, generating a representation describing the data pattern, and saving the data and the representation.

## Claims

### Claim 1

A method, implemented at least in part by one or more computing devices, for data storage in a collective parallel processing environment, the method comprising:
receiving data to be written for a plurality of collective processes within a collective parallel processing environment;
extracting a data pattern for the data to be written for the plurality of collective processes;
generating a representation describing the data pattern; and
saving the data and the representation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of collective processes are organized into a plurality of compute nodes, and wherein the data to be written is received from each compute node.

### Claim 3 (depends on 1)

The method of claim 1, wherein the data is structured data, wherein each collective process of the plurality of collective processes writes its respective data, and wherein the representation is a single index entry.

### Claim 4 (depends on 3)

The method of claim 3, wherein the data and the representation are saved using Parallel Log-structured File System (PLFS).

### Claim 5 (depends on 1)

The method of claim 1, wherein the data is unstructured data, wherein each collective process of the plurality of collective processes writes its respective data, and wherein the representation is a single index entry.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
before extracting the data pattern, shuffling data between neighboring collective processes of the plurality of collective processes such that the data pattern is more easily extracted.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
applying collective buffering techniques to create structured data across the plurality of collective processes.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining that data to be written for the plurality of collective processes is unstructured data;
based on the determining, shuffling data between neighboring collective processes to create structured data, wherein the data pattern is extracted based on the structured data.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
determining that data to be written for the plurality of collective processes is unstructured data;
based on the determining, shuffling data between neighboring compute nodes to create structured data, wherein the data pattern is extracted based on the structured data, and wherein the data is stored using PLFS in a number of physical files corresponding to the number of compute nodes.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining that data to be written for the plurality of collective processes is unstructured data;
based on the determining, shuffling data between neighboring collective processes to create structured data, wherein the structured data for each collective process has the same number of bytes or the same number of complex structures, except for a remainder which is written by a last collective process.

### Claim 11 (depends on 1)

The method of claim 1, wherein the plurality of collective processes are organized into a plurality of compute nodes, and wherein the data to be written is moved between neighboring compute nodes such that a same size of data to be written is received from each compute node, except for a last compute node from which a remainder of less than the same size of data may be received.

### Claim 12 (depends on 1)

The method of claim 1, wherein the representation comprises information describing a structure of the data across the plurality of collective processes.

### Claim 13

A computer-readable storage medium storing computer-executable instructions for causing a computing device to perform a method for data storage in a collective parallel processing environment, the method comprising:
receiving data to be written for a plurality of collective processes within a collective parallel processing environment;
extracting a data pattern for the data to be written for the plurality of collective processes;
generating a representation describing the data pattern; and
saving the data and the representation.

### Claim 14

A high-performance parallel processing computing environment comprising:
one or more computing devices comprising processing units and memory;
the one or more computing devices configured to perform operations comprising:
receiving data to be written for a plurality of collective processes within the parallel processing computing environment;
extracting a data pattern for the data to be written for the plurality of collective processes;
generating a representation describing the data pattern; and
saving the data and the representation, wherein the data and the representation are saved using Parallel Log-structured File System (PLFS).

### Claim 15 (depends on 14)

The high-performance parallel processing computing environment of claim 14, wherein the plurality of collective processes are organized into a plurality of compute nodes, and wherein the data to be written is received from each compute node.
