Method and apparatus with data loading
AssigneeSamsung Electronics Co., Ltd.
InventorsMyeong Woo Kim, Yongdeok Kim, Changin Choi, Seungwon Lee
Life of this patent
Estimating the term…
Abstract
A processor-implemented method with data loading includes: based on sizes of a plurality of data files in a training dataset, dividing the training dataset into a plurality of sub-sets; loading some data files in each sub-set into a plurality of processors; determining a packing combination of one or more data files loaded to processors in a same group among the plurality of processors, based on a ratio of a number of data files between the plurality of sub-sets and a batch size of distributed training; determining packed data files by packing the one or more data files according to the packing combination; and reallocating the packed data files to the processors in the same group.
Claims16
A processor-implemented method with data loading, the method comprising:
- based on sizes of a plurality of data files in a training dataset, dividing the training dataset into a plurality of sub-sets;
- loading some data files in each sub-set into a plurality of processors;
- determining a packing combination of one or more data files loaded to processors in a same group among the plurality of processors, based on a ratio of a number of data files between the plurality of sub-sets and a batch size of distributed training;
- determining packed data files by packing the one or more data files according to the packing combination;
- and
- reallocating the packed data files to the processors in the same group,
- wherein the reallocating of the packed data files comprises:
- aligning the packed data files in order of size;
- and
- in response to distributing a portion of the aligned packed data files to the processors in the same group in a predetermined first order, distributing another portion of the aligned packed data files in a second order which is a reverse order of the first order, and
- wherein the distributing of the other portion of the aligned packed data files in the second order in response to distributing the portion of the aligned packed data files in the first order is repeatedly performed in the batch size.
An apparatus with data loading, the apparatus comprising:
- one or more processors configured to:
- based on sizes of a plurality of data files in a training dataset, divide the training dataset into a plurality of sub-sets;
- load some data files in each sub-set into a plurality of processors;
- determine a packing combination of one or more data files loaded into processors in a same group among the plurality of processors, based on a ratio of a number of data files between the plurality of sub-sets and a batch size of distributed training;
- determine packed data files by packing the one or more data files according to the packing combination;
- and
- reallocate the packed data files to the processors in the same group,
- wherein, for the reallocating of the packed data files, the one or more processors are configured to:
- align the packed data files in order of size;
- and
- in response to distributing a portion of the aligned packed data files to the processors in the same group in a predetermined first order, distributing another portion of the aligned packed data files in a second order which is a reverse order of the first order, and
- wherein the one or more processors are configured to repeatedly perform the distributing of the other portion of the aligned packed data files in the second order in response to distributing the portion of the aligned packed data files in the first order in the batch size.
Dependencies are inferred from the claim text.