System and method for ontology induction through statistical profiling and reference schema matching

A system for use with a data integration or other computing environment, comprising:

• a knowledge source and a metadata stored therein and describing a plurality of data sources and data targets for use with data flow pipelines that receive data from selected data sources and provide the data to selected data targets;
• wherein each data flow includes a specification of the one or more data sources and data targets that operate as hubs and comprise datasets having attributes associated therewith,
• wherein a data flow is associated with actions that operate on one or more input datasets to transform and output data to one or more output datasets, and
• wherein a dataflow software application operates to receive input data from a source of data, and publish output data to one or more destinations, according to the data flow associated with the software application;
• wherein for a particular data source or data target, the system:
• accesses a schema descriptive of entity definitions within the particular data source or data target;
• profiles a sample data associated with the particular data source or data target for the one or more entities, based on one or more reference schemas;
• determines, in response to sampling the data associated with the particular data source or data target for the one or more entities, relationship rules that define associations:
• across datasets or entities within the particular data source or data target, or
• between the datasets or entities within the particular data source or data target and datasets or entities of others of the plurality of data sources and data targets;
• and
• automatically updates the knowledge source and the metadata stored therein to include an indication of the relationship rules determined across the datasets or entities, for use during development of the dataflow software application.

A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:

• accessing a knowledge source and a metadata stored therein and describing a plurality of data sources and data targets for use with data flow pipelines that receive data from selected data sources and provide the data to selected data targets;
• wherein each data flow includes a specification of the one or more data sources and data targets that operate as hubs and comprise datasets having attributes associated therewith,
• wherein a data flow is associated with actions that operate on one or more input datasets to transform and output data to one or more output datasets, and
• wherein a dataflow software application operates to receive input data from a source of data, and publish output data to one or more destinations, according to the data flow associated with the software application;
• for a particular data source or data target:
• accessing a schema descriptive of entity definitions within the particular data source or data target;
• profiling a sample data associated with the particular data source or data target for the one or more entities, based on one or more reference schemas;
• determining, in response to sampling the data associated with the particular data source or data target for the one or more entities, relationship rules that define associations:
• across datasets or entities within the particular data source or data target, or
• between the datasets or entities within the particular data source or data target and datasets or entities of others of the plurality of data sources and data targets;
• and
• automatically updating the knowledge source and the metadata stored therein to include an indication of the relationship rules determined across the datasets or entities, for use during development of the dataflow software application.