# US 2019/0171617 A1 · Method and system for flexible, high performance structured data processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171617 A1
- Document ID: US20190171617A1
- Status: Published application
- Kind: A1
- Application number: 16/313,830
- Filed: 2017-06-23
- Published: 2019-06-06
- Earliest claimed: 2016-06-28
- Assignees: Anditi Pty, Ltd.
- Applicants: Anditi Pty Ltd
- Inventors: Andrew Goodwin; Patrick Poissant; Shaileshkumar Lathiya; Peter Jamieson; Stephen Craig
- Web page: https://patentgrep.com/patent/US20190171617A1

## CPC

- G06F 16/13: File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06)
- G06F 16/258: {Data format conversion from or to a database}
- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 16/29: Geographical information databases
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Described herein is a method and system for flexible, high performance structured data processing. The method and system contains techniques for balancing and jointly optimising processing speed, resource utilisation, flexibility, scalability, and configurability in one workflow. A prime example of its application is the analysis of spatial data, e.g. LiDAR and imagery. However, the invention is applicable to a wide range of structured data problems in a variety of dimensions and settings.

## Claims

### Claim at position 1

-36. (canceled)

### Claim 37

A method using high performance computing for flexibly processing structured data that is stored as a dataset including a plurality of discrete data files, the method including:
accessing a data source where the structured dataset is stored;
pre-processing the dataset to generate:
(a) a reference file indicating one or more predetermined characteristics of the data that are contained within each of the discrete data files; and
(b) a metadata file indicating one or more data metrics; and
upon selection of data to process from the structured dataset:
defining plugin connections for processing the selection of data;
dynamically allocating data processing tasks to connected computer nodes; and
performing data processing on the selection of data; and
generating an output of the data processing of the selection of data.

### Claim 38 (depends on 37)

A method according to claim 37 wherein the step of pre-processing the dataset occurs in conjunction with indexing the dataset.

### Claim 39 (depends on 37)

A method according to claim 37 wherein the predetermined characteristics include data bounds and associated file names for each of the discrete data files in the dataset.

### Claim 40 (depends on 37)

A method according to claim 37, further including the step of pre-classifying the discrete data files including using discrete information from external files to calculate one or more data metrics.

### Claim 41 (depends on 40)

A method according to claim 40 wherein the data metrics include a determination of the likelihood of the presence of certain data features in discrete data files.

### Claim 42 (depends on 40)

A method according to claim 40 wherein the data metrics include measures of properties of the data set that will impact the computational complexity of each plugin in the analytical process.

### Claim 43 (depends on 39)

A method according to claim 39 wherein the step of pre-processing the dataset includes the steps of:
i) opening each discrete data file;
ii) determining the data bounds for each discrete data file; and
iii) storing the determined data bounds and an associated filename for each discrete data in the reference file.

### Claim 44 (depends on 43)

A method according to claim 43 wherein the structured dataset is a dataset of spatial and/or temporal data.

### Claim 45 (depends on 44)

A method according to claim 44 wherein the spatial data includes imagery and LiDAR data including point clouds derived there from. Preliminary Amendment

### Claim 46 (depends on 37)

A method according to claim 37, further including the step of performing automatic edge effect elimination on the selection of data, preferably by calculating a minimum required extra buffer of data at each plugin.

### Claim 47 (depends on 46)

A method according to claim 46 wherein the automatic edge effect elimination includes calculating, at each plugin, a total buffer of all downstream plugins.

### Claim 48 (depends on 46)

A method according to claim 46 wherein the plugin connections are polymorphic plugin connections, and wherein an output of at least one of the polymorphic plugin connections is preferably a derived type of the input to that polymorphic plugin connection

### Claim 49 (depends on 43)

A method according to claim 43, further including the steps of, upon receiving a request to process a selection of data from the structured dataset:
(a) reading the reference index;
(b) Adding a buffer to the requested area or time interval; and
(c) for a given data area or time interval indicative of the selection of data:
i) reading the data bounds corresponding to the given data area or time interval;
ii) comparing data bounds of the discrete data files with the data bounds of the given data area or time interval; and
iii) reading the discrete data files that at least partially overlap with the given data area or time interval.

### Claim 50 (depends on 49)

A method according to claim 49 wherein, upon receiving a request to process a work unit, comprising a selection of data and its associated buffer, a computer processor:
b1) reads the reference file;
b2) reads the data bounds corresponding to the work unit;
b3) compares data bounds of the discrete data files with the data bounds of the work unit; and
b4) reads the discrete data files that at least partially overlap with the data bounds of the work unit.

### Claim 51

A method of pre-classifying a dataset including a plurality of discrete data files distinguished by data bounds, the method including:
a1) creating a metadata file;
a2) opening the discrete data files;
a3) dividing the dataset into predefined data cells and determining at least one data metric for each of the data cells; and
a4) storing the at least one data metric in the metadata file in association with an associated data cell identifier for each data cell and an identifier of the discrete data file(s) associated with each data cell.

### Claim 52 (depends on 50)

A method according to claim 50 wherein at least one data metric includes a measure of likelihood that the data of an individual work unit includes specific spatial, temporal or spectral features.

### Claim 53 (depends on 50)

A method according to claim 50 wherein at least one data metric includes a measure of quality of the data within an individual work unit.

### Claim 54

A method of flexibly allocating high performance computer resources to a data processing operation on a dataset, the method including:
pre-processing the dataset to generate a metadata file including characteristics of the dataset;
dividing the dataset into a plurality of work units, each work unit indicative of a subset of the data contained in the dataset;
creating a list of work units of a predetermined size based on the characteristics of the dataset;
calculating the computational complexity of each work unit based on the size of work units and characteristics of the dataset;
determining memory requirements for each work unit;
determining available memory of connected computer nodes; and
allocating work units to connected computer nodes for processing based on available memory and the number of processes running.

### Claim 55 (depends on 54)

A method according to claim 54 including the step of merging or subdividing work units based on available memory of one or more connected computer nodes.
