# US 2017/0286509 A1 · Dynamic distributed generation of data representations from highly condensed data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286509 A1
- Document ID: US20170286509A1
- Status: Published application
- Kind: A1
- Application number: 15/474,832
- Filed: 2017-03-30
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: SAS Institute Inc.
- Applicants: SAS Institute Inc.
- Inventors: Stacey Michelle Christian; Michael Stephen Whitcher; Donald Kent McAlister; Phillip Elliot Hanna
- Web page: https://patentgrep.com/patent/US20170286509A1

## CPC

- G06F 16/283: {Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP}
- G06F 16/2264: {Multidimensional index structures}
- G06F 16/24532: {of parallel queries}
- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 16/1744: {using compression, e.g. sparse files}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An apparatus includes a processor and storage storing instructions causing the processor to store, at a node device of a grid of node devices, a data block of a data blob, receive data blob metadata indicative of an organization of data items within the data blob, receive a command to generate, from the data block, at least one row of a representation portion of a plurality of representation portions from which a 2D representation of the data blob is to be generated, use the data blob metadata and a node block map indicative of which data items of the data block are required to generate the representation portion to derive one or more transforms to be performed with the data block to generate the at least one row of the representation portion, and perform the one or more transforms with the data block to generate the at least one row.

## Claims

### Claim 1

An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:
store, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein:
the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;
the data blob comprises multiple data items; and
each data block of the plurality of data blocks comprises a subset of the multiple data items;
receive, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;
receive, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein:
a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;
the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and
the subset of node devices comprises the node device;
use the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein:
the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and
the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion; and
perform, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the processor is caused to perform operations comprising:
store, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by the processor to perform a different type of transform from others of the multiple reader routines;
select one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and
use the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the processor is caused to perform operations comprising:
transmit, from the node device to a coordinating device, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;
receive, at the node device, a command to store the data block from the coordinating device; and
receive, at the node device, the data block from the at least one source device.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the processor is caused to perform operations comprising:
receive, at the node device, the command to generate at least one row of the representation portion from the coordinating device; and
transmit, from the node device, the at least one row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one row of the representation data from at least one row of each representation portion of the plurality of representation portions.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the processor is caused to perform operations comprising:
receive, at the node device, at least a portion of the representation data from the coordinating device;
receive, at the node device, a command from the coordinating device to store the portion of the representation data; and
store, at the node device, the received portion of the representation data.

### Claim 6 (depends on 3)

The apparatus of claim 3, wherein the processor is caused to perform operations comprising:
receive, at the node device, another command to generate at least one more row of the representation portion from the data block;
retrieve first state data indicative of a state of the node device after the performance of the one or more transforms to generate the at least one row of the representation portion;
use the first state data to prepare the node device for another performance of the one or more transforms with the data block to generate the at least one more row of the representation portion;
store, at the node device, second state data indicative of a state of the node device after the another performance of the one or more transforms to generate the at least one more row of the representation portion; and
transmit, from the node device, the at least one more row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one more row of the representation data from at least one more row of each representation portion of the plurality of representation portions.

### Claim 7 (depends on 3)

The apparatus of claim 3, wherein the processor is caused to perform operations comprising:
use the node bock map and an identifier of the requesting device or an entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and
in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refrain from using the portion of the data items to generate the at least one row of the representation portion.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein:
the node device comprises a coordinating device of the grid of node devices to coordinate the generation of at least one row of each representation portion by the subset of node devices; and
the processor is caused to perform operations comprising:
receive, at the coordinating device, the command from the requesting device; and
in response to the receipt of the command from the requesting device, perform operations comprising:
generate the plurality of node block maps from a master block map that comprises an indication of distribution of the multiple data items within the data blob among the plurality of data blocks, and an indication of the distribution of the plurality of data blocks among the grid of node devices; and
distribute, from the coordinating device, the plurality of node block maps among the subset of node devices.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the processor is caused to perform operations comprising use representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein a transform of the one or more transforms is selected from a group consisting of aggregation of data items of the data blob, averaging of data items of the data blob, sorting data items of the data blob, calculating a value of at least one column of the at least one row from a value of at least one other column of the at least one row, decompressing data items of the data blob, and decrypting data items of the data blob.

### Claim 11

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:
store, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein:
the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;
the data blob comprises multiple data items; and
each data block of the plurality of data blocks comprises a subset of the multiple data items;
receive, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;
receive, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein:
a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;
the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and
the subset of node devices comprises the node device;
use the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein:
the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and
the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion; and
perform, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion.

### Claim 12 (depends on 11)

The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:
store, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by the processor to perform a different type of transform from others of the multiple reader routines;
select one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and
use the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

### Claim 13 (depends on 11)

The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:
transmit, from the node device to a coordinating device, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;
receive, at the node device, a command to store the data block from the coordinating device; and
receive, at the node device, the data block from the at least one source device.

### Claim 14 (depends on 13)

The computer-program product of claim 13, wherein the processor is caused to perform operations comprising:
receive, at the node device, the command to generate at least one row of the representation portion from the coordinating device; and
transmit, from the node device, the at least one row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one row of the representation data from at least one row of each representation portion of the plurality of representation portions.

### Claim 15 (depends on 14)

The computer-program product of claim 14, wherein the processor is caused to perform operations comprising:
receive, at the node device, at least a portion of the representation data from the coordinating device;
receive, at the node device, a command from the coordinating device to store the portion of the representation data; and
store, at the node device, the received portion of the representation data.

### Claim 16 (depends on 13)

The computer-program product of claim 13, wherein the processor is caused to perform operations comprising:
receive, at the node device, another command to generate at least one more row of the representation portion from the data block;
retrieve first state data indicative of a state of the node device after the performance of the one or more transforms to generate the at least one row of the representation portion;
use the first state data to prepare the node device for another performance of the one or more transforms with the data block to generate the at least one more row of the representation portion;
store, at the node device, second state data indicative of a state of the node device after the another performance of the one or more transforms to generate the at least one more row of the representation portion; and
transmit, from the node device, the at least one more row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one more row of the representation data from at least one more row of each representation portion of the plurality of representation portions.

### Claim 17 (depends on 13)

The computer-program product of claim 13, wherein the processor is caused to perform operations comprising:
use the node bock map and an identifier of the requesting device or a entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and
in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refrain from using the portion of the data items to generate the at least one row of the representation portion.

### Claim 18 (depends on 11)

The computer-program product of claim 11, wherein:
the node device comprises a coordinating device of the grid of node devices to coordinate the generation of at least one row of each representation portion by the subset of node devices; and
the processor is caused to perform operations comprising:
receive, at the coordinating device, the command from the requesting device; and
in response to the receipt of the command from the requesting device, perform operations comprising:
generate the plurality of node block maps from a master block map that comprises an indication of distribution of the multiple data items within the data blob among the plurality of data blocks, and an indication of the distribution of the plurality of data blocks among the grid of node devices; and
distribute, from the coordinating device, the plurality of node block maps among the subset of node devices.

### Claim 19 (depends on 18)

The computer-program product of claim 18, wherein the processor is caused to perform operations comprising use representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

### Claim 20 (depends on 18)

The computer-program product of claim 18, wherein the at least one operation is selected from a group consisting of aggregation of values of the at least one representation portion, averaging of data items of the at least one representation portion, sorting data items of the at least one representation portion, calculating a value of at least one column of the at least one row from a value of at least one other column of the at least one row, and transmitting the at least one row of the representation data to the grid of node devices for storage.

### Claim 21

A computer-implemented method comprising:
storing, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein:
the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;
the data blob comprises multiple data items; and
each data block of the plurality of data blocks comprises a subset of the multiple data items;
receiving, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;
receiving, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein:
a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;
the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and
the subset of node devices comprises the node device;
using the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein:
the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and
the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion; and
performing, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, comprising:
storing, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by a processor of the node device to perform a different type of transform from others of the multiple reader routines;
selecting one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and
using the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, comprising:
transmitting, from the node device to a coordinating device, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;
receiving, at the node device, a command to store the data block from the coordinating device; and
receiving, at the node device, the data block from the at least one source device.

### Claim 24 (depends on 23)

The computer-implemented method of claim 23, comprising:
receiving, at the node device, the command to generate at least one row of the representation portion from the coordinating device; and
transmitting, from the node device, the at least one row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one row of the representation data from at least one row of each representation portion of the plurality of representation portions.

### Claim 25 (depends on 24)

The computer-implemented method of claim 24, comprising:
receiving, at the node device, at least a portion of the representation data from the coordinating device;
receiving, at the node device, a command from the coordinating device to store the portion of the representation data; and
storing, at the node device, the received portion of the representation data.

### Claim 26 (depends on 23)

The computer-implemented method of claim 23, comprising:
receiving, at the node device, another command to generate at least one more row of the representation portion from the data block;
retrieving first state data indicative of a state of the node device after the performance of the one or more transforms to generate the at least one row of the representation portion;
using the first state data to prepare the node device for another performance of the one or more transforms with the data block to generate the at least one more row of the representation portion;
storing, at the node device, second state data indicative of a state of the node device after the another performance of the one or more transforms to generate the at least one more row of the representation portion; and
transmitting, from the node device, the at least one more row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one more row of the representation data from at least one more row of each representation portion of the plurality of representation portions.

### Claim 27 (depends on 23)

The computer-implemented method of claim 23, comprising:
using the node bock map and an identifier of the requesting device or a entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and
in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refraining from using the portion of the data items to generate the at least one row of the representation portion.

### Claim 28 (depends on 21)

The computer-implemented method of claim 21, wherein:
the node device comprises a coordinating device of the grid of node devices to coordinate the generation of at least one row of each representation portion by the subset of node devices; and
the method comprises:
receiving the command from the requesting device; and
in response to the receipt of the command from the requesting device, performing operations comprising:
generating the plurality of node block maps from a master block map that comprises an indication of distribution of the multiple data items within the data blob among the plurality of data blocks, and an indication of the distribution of the plurality of data blocks among the grid of node devices; and
distributing, from the coordinating device, the plurality of node block maps among the subset of node devices.

### Claim 29 (depends on 28)

The computer-implemented method of claim 28, comprising using representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

### Claim 30 (depends on 21)

The computer-implemented method of claim 21, wherein:
the data blob comprises a hypercube that employs an indexing scheme of at least 3 dimensions; and
the data blob does not comprise the 2D representation of data provided by the representation data as a slice of the hypercube.
