Executing a big data analytics pipeline using shared storage resources
AssigneePure Storage, Inc.
InventorsIvan Jibaja, Stefan Dorsett, Prashant Jaikumar, Roy Kim, Curtis Pullen
Life of this patent
Estimating the term…
Abstract
Executing a big data analytics pipeline in a storage system that includes compute resources and shared storage resources, including: receiving, from a data producer, a dataset; storing, within the storage system, the dataset; allocating processing resources to an analytics application; and executing the analytics application on the processing resources, including ingesting the dataset from the storage system.
Claims17
A method comprising:
- executing an analytics application on first compute resources of a storage system, including reading by the analytics application at least a first portion of a structure dataset converted, by the storage system, from an unstructured dataset received, by the storage system, from a data producer, wherein the at least a first portion of the structured dataset is read from a location determined by storage system to use to store the structured dataset that is specific to the data producer;
- detecting that the analytics application has ceased executing properly on the first compute resources;
- and
- executing the analytics application on second compute resources on the storage system responsive to detecting that the analytics application has ceased executing properly at the storage system, including reading by the analytics application at least a second portion of the structured dataset from the location within the storage system that is specific to the data producer such that the structured dataset is not copied and is not sent over a network connection in response to the reading at least a second portion of the structured dataset.
An apparatus including a computer processor and a computer memory, the computer memory including computer program instructions that, when executed, cause the apparatus to carry out the steps of:
- executing an analytics application on first compute resources of a storage system, including reading by the analytics application at least a first portion of a structure dataset converted, by the storage system, from an unstructured dataset received, by the storage system, from a data producer, wherein the at least a first portion of the structured dataset is read from a location determined by storage system to use to store the structured dataset that is specific to the data producer;
- detecting that the analytics application has ceased executing properly on the first compute resources;
- and
- executing the analytics application on second compute resources on the storage system responsive to detecting that the analytics application has ceased executing properly at the storage system, including reading by the analytics application at least a second portion of the structured dataset from the location within the storage system that is specific to the data producer such that the structured dataset is not copied and is not sent over a network connection in response to the reading at least a second portion of the structured dataset.
A computer program product disposed on a non-transitory computer readable storage medium, the computer program product including computer program instructions that, when executed, cause a computer to carry out the steps of:
- executing an analytics application on first compute resources of a storage system, including reading by the analytics application at least a first portion of a structure dataset converted, by the storage system, from an unstructured dataset received, by the storage system, from a data producer, wherein the at least a first portion of the structured dataset is read from a location determined by storage system to use to store the structured dataset that is specific to the data producer;
- detecting that the analytics application has ceased executing properly on the first compute resources;
- and
- executing the analytics application on second compute resources on the storage system responsive to detecting that the analytics application has ceased executing properly at the storage system, including reading by the analytics application at least a second portion of the structured dataset from the location within the storage system that is specific to the data producer such that the structured dataset is not copied and is not sent over a network connection in response to the reading at least a second portion of the structured dataset.
Dependencies are inferred from the claim text.