Ensuring compliance of data for use in a data pipeline with limitations on data collection

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing operation of a data pipeline, the operations comprising:

• obtaining data comprising a populated field and an unpopulated field, the unpopulated field lacking information due to limitations on information collection;
• making a determination regarding whether a first inference model can predict at least the unpopulated field within a reliability range, the reliability range ensuring that inferences generated by the first inference model comply with the limitations;
• in an instance of the determination in which the first inference model can predict the at least the unpopulated field:
• generating an inference using the first inference model;
• populating the unpopulated field using the inference to obtain supplemented data;
• and
• providing the supplemented data to a downstream consumer, wherein
• the operations further comprise, prior to obtaining the data:
• using a second inference model to qualify which fields of the data are predictable using other fields of the data;
• and
• performing a training process based on the fields of the data that are predictable to obtain the first inference model, wherein the first inference model is trained based on qualified training data comprising a subset of all available training data, the subset being selected based on the reliability range.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing operation of a data pipeline, the operations comprising:
• obtaining data comprising a populated field and an unpopulated field, the unpopulated field lacking information due to limitations on information collection;
• making a determination regarding whether a first inference model can predict at least the unpopulated field within a reliability range, the reliability range ensuring that inferences generated by the first inference model comply with the limitations;
• in an instance of the determination in which the first inference model can predict the at least the unpopulated field:
• generating an inference using the first inference model;
• populating the unpopulated field using the inference to obtain supplemented data;
• and
• providing the supplemented data to a downstream consumer, wherein
• the operations further comprise, prior to obtaining the data:
• using a second inference model to qualify which fields of the data are predictable using other fields of the data;
• and
• performing a training process based on the fields of the data that are predictable to obtain the first inference model, wherein the first inference model is trained based on qualified training data comprising a subset of all available training data, the subset being selected based on the reliability range.