Back to results
US 12,008,404 B2Granted
Executing a big data analytics pipeline using shared storage resources
AssigneePure Storage, Inc.
InventorsIvan Jibaja, Prashant Jaikumar, Stefan Dorsett, Curtis Pullen, Roy Kim
Life of this patent
Estimating the term…
Abstract
Executing a big data analytics pipeline in a storage system that includes compute resources and shared storage resources, including: receiving, from a data producer, a dataset; storing, within the storage system, the dataset; allocating processing resources to an analytics application; and executing the analytics application on the processing resources, including ingesting the dataset from the storage system.
Claims20
A method comprising:
- receiving a dataset by polling a location to which a data producer writes the dataset;
- converting, by a storage system, the dataset for use by an analytics application;
- storing, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer;
- executing the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources;
- detecting that the analytics application has ceased executing properly on the first compute resources;
- and
- in response to detecting that the analytics application has ceased executing properly on the first compute resources, executing the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location of the shared storage resources that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources.
4 more elements
An apparatus comprising:
- a memory;
- and
- a processing device, operatively coupled to the memory, configured to:
- receive a dataset by polling a location to which a data producer writes the dataset;
- convert, by a storage system, the dataset for use by an analytics application;
- storing, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer
- execute the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources;
- detect that the analytics application has ceased executing properly on the first compute resources;
- and
- in response to detecting that the analytics application has ceased executing properly, execute the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources.
7 more elements
A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to:
- receive a dataset by polling a location to which a data producer writes the dataset;
- convert, by a storage system, the dataset for use by an analytics application;
- store, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer;
- execute the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources;
- detect that the analytics application has ceased executing properly on the first compute resources;
- and
- in response to detecting that the analytics application has ceased executing properly on the first compute resources, execute the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location of the shared storage resources that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources.
4 more elements
Dependencies are inferred from the claim text.