Batch building for deep learning training workloads
AssigneePure Storage, Inc.
InventorsEmily Potyraj, Brian Gold
Life of this patent
Estimating the term…
Abstract
Batch building for artificial intelligence workflows, including: issuing, responsive to a request for a batch of data objects, requests to a data repository for multiple data objects stored among one or more directories; selecting, in accordance with a batch building policy, a subset of data objects based on one or more responses to the requests; and providing, to the artificial intelligence workflow, a batch of data objects that includes the subset of data objects.
Claims20
A method of batch building for artificial intelligence workflows, the method comprising:
- issuing, by an artificial intelligence and machine learning infrastructure system responsive to a request from a deep learning computing system for a batch of data objects, requests to a data storage system for multiple data objects stored among one or more directories;
- selecting, by the artificial intelligence and machine learning infrastructure system in accordance with a batch building policy, a subset of data objects stored in the data storage system based on one or more responses to the requests, wherein the batch building policy describes a sequence of data types, and wherein the one or more responses comprise characteristics of files within a file system of the data storage system;
- and
- providing, to the deep learning computing system by the artificial intelligence and machine learning infrastructure system responsive to the request for the batch of data objects, one or more data objects that include the subset of data objects in the sequence described by the batch building policy.
An artificial intelligence and machine learning infrastructure system comprising:
- one or more storage systems comprising, respectively, one or more storage devices;
- and
- one or more graphical processing units, wherein the graphical processing units are configured to communicate with the one or more storage systems over a communication fabric;
- wherein the artificial intelligence and machine learning infrastructure system is configured to:
- issue, responsive to a request for a batch of data objects from a deep learning computing system, requests to a data storage system for multiple data objects stored among one or more directories;
- select, in accordance with a batch building policy, a subset of data objects stored in the data storage system based on one or more responses to the requests, wherein the batch building policy describes a sequence of data types, and wherein the one or more responses comprise characteristics of files within a file system of the data storage system;
- and
- provide, to the deep learning computing system responsive to the request for the batch of data objects, one or more data objects that include the subset of data objects in the sequence described by the batch building policy.
Dependencies are inferred from the claim text.