# US 12,008,404 B2 · Executing a big data analytics pipeline using shared storage resources > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,404 B2 - Document ID: US12008404B2 - Status: Granted - Kind: B2 - Application number: 17/721,175 - Filed: 2022-04-14 - Published: 2022-07-28 - Granted: 2024-06-11 - Earliest claimed: 2017-10-19 - Assignees: Pure Storage, Inc. - Applicants: Pure Storage, Inc. - Inventors: Ivan Jibaja; Prashant Jaikumar; Stefan Dorsett; Curtis Pullen; Roy Kim - Web page: https://patentgrep.com/patent/US12008404B2 ## CPC - G06F 9/5005: {to service a request} - G06F 9/5011: {the resources being hardware resources other than CPUs, Servers and Terminals} - G06F 3/061: {Improving I/O performance} - G06F 3/0631: {by allocating resources to storage systems} - G06F 3/0665: {at area level, e.g. provisioning of virtual or logical volumes} - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 9/4856: {resumption being on a different machine, e.g. task migration, virtual machine migration (G06F9/5088 takes precedence)} - G06F 9/505: {considering the load} - G06F 9/5088: {involving task migration} - G06F 16/2272: {Management thereof} - G06F 16/258: {Data format conversion from or to a database} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Executing a big data analytics pipeline in a storage system that includes compute resources and shared storage resources, including: receiving, from a data producer, a dataset; storing, within the storage system, the dataset; allocating processing resources to an analytics application; and executing the analytics application on the processing resources, including ingesting the dataset from the storage system. ## Claims ### Claim 1 A method comprising: receiving a dataset by polling a location to which a data producer writes the dataset; converting, by a storage system, the dataset for use by an analytics application; storing, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer; executing the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources; detecting that the analytics application has ceased executing properly on the first compute resources; and in response to detecting that the analytics application has ceased executing properly on the first compute resources, executing the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location of the shared storage resources that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources. ### Claim 2 (depends on 1) The method of claim 1, wherein the first compute resources and the second compute resources are located within a physical storage system. ### Claim 3 (depends on 1) The method of claim 1, further comprising: allocating additional processing resources to a real-time analytics application; and executing the real-time analytics application on the additional processing resources, including ingesting at least a portion of the dataset. ### Claim 4 (depends on 1) The method of claim 1, further comprising: receiving, by a storage system, an unstructured dataset; converting, by the storage system, the unstructured dataset into a structured dataset; and storing the structured dataset within the storage system. ### Claim 5 (depends on 1) The method of claim 1, further comprising: detecting that the analytics application needs additional processing resources; allocating additional processing resources to the analytics application; and executing the analytics application on the additional processing resources. ### Claim 6 (depends on 1) The method of claim 1, wherein the dataset includes log files describing one or more execution states of a computing system and wherein executing the analytics application on the compute resources further comprises: evaluating the log files to identify one or more execution patterns associated with the computing system. ### Claim 7 (depends on 6) The method of claim 6, wherein evaluating the log files to identify one or more execution patterns associated with the computing system further comprises: comparing fingerprints associated with known execution patterns to information contained in the log files. ### Claim 8 An apparatus comprising: a memory; and a processing device, operatively coupled to the memory, configured to: receive a dataset by polling a location to which a data producer writes the dataset; convert, by a storage system, the dataset for use by an analytics application; storing, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer execute the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources; detect that the analytics application has ceased executing properly on the first compute resources; and in response to detecting that the analytics application has ceased executing properly, execute the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the first compute resources and the second compute resources are located within a physical storage system. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the processing device is further configured to: allocate additional processing resources to a real-time analytics application; and execute the real-time analytics application on the additional processing resources, including ingesting at least a portion of the dataset. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the processing device is further configured to: receive, by a storage system, an unstructured dataset; convert, by the storage system, the unstructured dataset into a structured dataset; and store the structured dataset within the storage system. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein the processing device is further configured to: detect that the analytics application needs additional processing resources; allocate additional processing resources to the analytics application; and execute the analytics application on the additional processing resources. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the dataset includes log files describing one or more execution states of a computing system and wherein to execute the analytics application on the compute resources, the processing device is further configured to: evaluate the log files to identify one or more execution patterns associated with the computing system. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein to evaluate the log files to identify one or more execution patterns associated with the computing system, the processing device is further configured to: compare fingerprints associated with known execution patterns to information contained in the log files. ### Claim 15 A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to: receive a dataset by polling a location to which a data producer writes the dataset; convert, by a storage system, the dataset for use by an analytics application; store, by the storage system, the dataset within shared storage resources of the storage system at a particular location specific to the data producer; execute the analytics application on first compute resources of the storage system to ingest a first portion of the dataset that is read from the particular location of the shared storage resources; detect that the analytics application has ceased executing properly on the first compute resources; and in response to detecting that the analytics application has ceased executing properly on the first compute resources, execute the analytics application on second compute resources of the storage system to ingest a second portion of the dataset from the particular location of the shared storage resources that is specific to the data producer while bypassing copying the second portion of the dataset to a different location of the shared storage resources. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the first compute resources and the second compute resources are located within a physical storage system. ### Claim 17 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the processing device is further configured to: allocate additional processing resources to a real-time analytics application; and execute the real-time analytics application on the additional processing resources, including ingesting at least a portion of the dataset. ### Claim 18 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the processing device is further configured to: receive, by a storage system, an unstructured dataset; convert, by the storage system, the unstructured dataset into a structured dataset; and store the structured dataset within the storage system. ### Claim 19 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the processing device is further configured to: detect that the analytics application needs additional processing resources; allocate additional processing resources to the analytics application; and execute the analytics application on the additional processing resources. ### Claim 20 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the dataset includes log files describing one or more execution states of a computing system and wherein to execute the analytics application on the compute resources the processing device is further configured to: evaluate the log files to identify one or more execution patterns associated with the computing system.