Method and system for managing and using data confidence in a decentralized computing platform

A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:

• performing, by a cleaning module, a cleaning process on a local data set to generate a confidence score corresponding to the local data set, wherein a local data system comprises a local data manager, a local data source, and the cleaning module, wherein the local data set is stored on the local data system;
• sending, by the cleaning module, a data set entry to a catalog node, wherein the data set entry comprises the confidence score, local data set metadata, and cleaning module metadata,
• wherein the local data set metadata comprises at least one of:
• a geographical location of the local data source, a type of the local data source, and the local data manager;
• obtaining, by a virtual machine (VM) executing on a computing node, a data computation request from a client;
• in response to the data computation request:
• sending a metadata request to the catalog node based on the data computation request;
• obtaining a plurality of data set entries from the catalog node based on the metadata request, wherein the plurality of data set entries comprises at least the data set entry;
• performing a confidence score analysis on the plurality of data set entries to obtain selected data set entries, wherein the selected data set entries comprise at least the data set entry;
• initiating, on the local data system, a data computation using the local data set;
• obtaining, in response to the initiating, a result from the local data system;
• and
• sending a data computation result to the client based on the data computation, wherein the data computation result is based on the result.

A system, comprising:

• a processor;
• and
• memory comprising instructions, which when executed by the processor, perform a method, the method comprising:
• performing, by a cleaning module, a cleaning process on a local data set to generate a confidence score corresponding to the local data set, wherein a local data system comprises a local data manager, a local data source, and the cleaning module, wherein the local data set is stored on the local data system;
• sending, by the cleaning module, a data set entry to a catalog node, wherein the data set entry comprises the confidence score, local data set metadata, and cleaning module metadata,
• wherein the local data set metadata comprises at least one of:
• a geographical location of the local data source, a type of the local data source, and the local data manager;
• obtaining, by a virtual machine (VM) executing on a computing node, a data computation request from a client;
• in response to the data computation request:
• sending a metadata request to the catalog node based on the computation request;
• obtaining a plurality of data set entries from the catalog node based on the metadata request, wherein the plurality of data set entries comprises at least the data set entry;
• performing a confidence score analysis on the plurality of data set entries to obtain selected data set entries, wherein the selected data set entries comprise at least the data set entry;
• initiating, on the local data system, a data computation using the local data set;
• obtaining, in response to the initiating, a result from the local data system;
• and
• sending a data computation result to the client based on the data computation, wherein the data computation result is based on the result.