Ensuring data integrity during large-scale data migration

A system, comprising:

• one or more processors;
• and
• memory including instructions that, as a result of being executed by the one or more processors, cause the system to:
• obtain, by a first host, a first message from a first message queue, wherein the first message comprises a first storage location of an untransformed data object, wherein the untransformed data object is stored in the first storage location of a data storage service of a computing resource service provider;
• transform, by the first host, the untransformed data object into a first set of transformed data according to a transformation scheme;
• generate, by the first host, a second message for a second message queue, wherein the second message comprises the first message and an error detection code associated with the first set of transformed data;
• obtain, by a second host, the second message from the second message queue;
• parse, by the second host, the second message to determine the first storage location of the untransformed data object indicated by the first message;
• transform, by the second host, the untransformed data object into a second set of transformed data according to the transformation scheme;
• verify, by the second host, the second set of transformed data based at least in part on the error detection code associated with the first set of transformed data, wherein the error detection code is obtained from the second message, and wherein the verification is performed based at least in part on a determination that a computer system that generated the second set of transformed data is different from a computer system that generated the error detection code;
• and
• as a result of the verification by the second host, cause the second set of transformed data to be stored in a second storage location different from the first storage location.

A non-transitory computer-readable storage medium having collectively stored thereon executable instructions that, as a result of being executed by one or more processors of a set of computer systems, cause the set of computer systems to at least:

• obtain, by a first host, a first message from a set of messages, wherein the first message comprises a location of a first data store comprising an untransformed data object, wherein the first data store is hosted by a data storage service of a computing resource service provider;
• obtain, by the first host, the untransformed data object from the first data store indicated by the first message;
• generate, by the first host, a second message comprising at least the first message and error detection codes corresponding to a transformation of the untransformed data object into one or more transformed data objects according to a transformation scheme;
• provide, by the first host, the second message to the set of messages;
• obtain, by a second host, the untransformed data object associated with the error detection codes from the first data store based at least in part on information obtained in the second message;
• transform, by the second host, the untransformed data object according to the transformation scheme into a data transformation;
• verify, by the second host, the data transformation against the error detection codes corresponding to the transformation of the untransformed data object according to the transformation scheme, wherein the error detection code is obtained from the second message, and wherein the verification is performed based at least in part on a determination that a computer system that generated the data transformation is different from a computer system that generated the error detection codes;
• and
• as a result of the verification by the second host, store the data transformation in a second data store different from the first data store.