Converting a data stream into files

A method of processing a plurality of files from a data stream, comprising:

• initiating, by a device, a plurality of worker nodes supported by a cloud computing system, wherein a worker node in the plurality of worker nodes is associated with a unique partition in a plurality of partitions,
• wherein a key is determined for a partition of the plurality of partitions,
• wherein the key comprises a hash that is determined for an event in the data stream based on an identifier associated with an object related to the event, and
• wherein a plurality of events, including the event, associated with a same object is assigned to a same key and a same partition;
• verifying, by the device, that the worker node has been initiated based on a file received indicating a start of the worker node;
• querying, by the device, and based on transmitting a request that includes the key, a database storing the data stream;
• receiving, by the device, a portion of the data stream associated with the unique partition corresponding to the worker node;
• converting, by the device, the portion of the data stream into a legacy format version;
• encoding, by the device, the legacy format version of the portion of the data stream;
• uploading, by the device, a corresponding file, out of the plurality of files, that corresponds to encoded data from encoding the legacy format version of the portion of the data stream;
• uploading, by the device, a done file based on the plurality of worker nodes uploading the plurality of files;
• detecting, by the device, the done file in a remote storage;
• determining, by the device, the plurality of files based on the done file;
• receiving, by the device, the plurality of files;
• and
• processing, by the device, the plurality of files in sequence to update a set of objects.

A non-transitory computer-readable medium storing a set of instructions for converting a data stream into a plurality of files, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a device, cause the device to:
• initiate a plurality of worker nodes supported by a cloud computing system and associated with a plurality of partitions,
• wherein a key is determined for a partition of the plurality of partitions,
• wherein the key comprises a hash that is determined for an event in the data stream based on an identifier associated with an object related to the event,
• wherein a plurality of events, including the event, associated with a same object is assigned to a same key and a same partition, and
• wherein a worker node of the plurality of work nodes is associated with a partition in the plurality of partitions;
• query a database storing the data stream;
• receive a portion of the data stream associated with one or more partitions, in the plurality of partitions, corresponding to the worker node;
• convert the data stream into legacy format versions;
• encode a portion of the legacy format versions;
• upload the plurality of files, wherein each file in the plurality of files corresponds to encoded data from encoding the portion of the legacy format versions;
• and
• upload a done file based on uploading the plurality of files.