Distributed data framework for data analytics

A computer readable non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to perform the steps of:

• loading, by a data analysis system, a data set from a distributed file system, wherein the data set is distributed across a plurality of files;
• storing, by the data analysis system, the data set as an in-memory distributed data structure across a plurality of processors, wherein the data analysis system marks the distributed data structure as immutable;
• receiving a request from a first client device, the request identifying the in-memory distributed data structure;
• responsive to receiving the request from the first client device, generating, by the data analysis system, a uniform resource identifier (URI) uniquely identifying the in-memory distributed data structure within the data analysis system;
• storing the generated URI in a mapping from generated URIs to locations of in-memory distributed data structures;
• providing the URI to the first client device;
• receiving, by the data analysis system, from a second client device, a request specifying the generated URI, wherein the second client device received the generated URI from the first client device via a communication mechanism outside the data analysis system;
• converting the distributed data structure from immutable to mutable;
• executing the request received from the second client device by one or more processors from the plurality of processors storing the in-memory distributed data structure;
• and
• returning the result of execution of the request to the second client device.

A computer-implemented system for processing of distributed data structures, the system comprising:

• a plurality of processors;
• and
• a computer readable non-transitory storage medium storing instructions thereon, the instructions for performing the steps of:
• loading, by a data analysis system, a data set from a distributed file system, wherein the data set is distributed across a plurality of files;
• storing, by the data analysis system, the data set as an in-memory distributed data structure across the plurality of processors, wherein the data analysis system marks the distributed data structure as immutable;
• receiving a request from a first client device, the request identifying the in-memory distributed data structure;
• responsive to receiving the request from the first client device, generating, by the data analysis system, a uniform resource identifier (URI) uniquely identifying the in-memory distributed data structure within the data analysis system;
• storing the generated URI in a mapping from generated URIs to locations of in-memory distributed data structures;
• providing the URI to the first client device;
• receiving, by the data analysis system, from a second client device, a request specifying the generated URI, wherein the second client device received the generated URI from the first client device via a communication mechanism outside the data analysis system;
• converting the distributed data structure from immutable to mutable;
• executing the request received from the second client device by one or more processors from the plurality of processors storing the in-memory distributed data structure;
• and
• returning the result of execution of the request to the second client device.