Methods, apparatus and computer programs for evaluating and using a resilient data representation

A method to classify data from a plurality of semi-structured data sources, said method comprising:

• parsing a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• for a set of candidate generalized representative labels that each represent a set of instances of a data element, estimating a probability of each respective candidate generalized representative label satisfying a set of resilience criteria following potential structural changes to the semi-structured data source, wherein a resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source;
• comparing the estimated probabilities to identify a generalized representative label having a highest probability of satisfying the set of resilience criteria;
• selecting the identified generalized representative lable;
• and
• storing the selected identified generalized representative label,
• wherein said estimating occurs by determining a probability that said respective candidate generalized label will match nodes of interest in said DOM tree and determining a probability that said respective candidate generalized label will match nodes not of interest in said DOM tree, and
• wherein said Identified generalized representative label is used as a measure to classify data from said plurality of semi-structured data sources.

A method to cluster data from a plurality of semi-structured data sources, said method comprising:

• parsing a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• generating a respective representative label for each of a set of identified instances of a data element within said semi-structured data source;
• generating a generalized representative label that matches each of the representative labels generated for the identified instances;
• evaluating a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• storing the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to cluster data from said plurality of semi-structured data sources.

A method for extraction of data elements from a semi-structured data source, said method comprising:

• parsing said semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• for at least one candidate representative label for representing data elements within the data source, evaluating resilience of the candidate representative label to structural changes of the data source by estimating, for a set of possible structural changes to the data source, a probability of the candidate representative label satisfying a set of resilience criteria;
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said estimating occurs by determining a probability that said candidate representative label will match nodes of interest in said DOM tree and determining a probability that said candidate representative label will match nodes not of interest in said DOM tree,
• selecting a representative label based on the estimated probability of satisfying the set of resilience criteria;
• storing the representative label;
• and
• executing a data extraction rule using the selected representative label to specify data to be extracted, wherein a resilience of said selected representative label is used as a measure to extract data from said plurality of semi-structured data sources.

A program storage device readable by computer, embodying a program of instructions executable by said computer to perform a method to extract data from a plurality of semi-structured data sources, said method comprising:

• parsing a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• generating a respective representative label for each of a set of identified instances of a data element within said semi-structured data source;
• generating a generalized representative label that matches each of the representative labels generated for the identified instances;
• evaluating a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• storing the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said evaluating comprises evaluating the resilience against a selection criterion by:
• comparing a calculated degree of resilience for the generalized representative label with a calculated degree of resilience of another generalized representative label;
• and
• selecting the generalized representative label having a highest calculated degree of resilience,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to extract data from said plurality of semi-structured data sources.

A program storage device readable by computer, embodying a program of instructions executable by said computer to perform a method to classify data from a plurality of semi-structured data sources, said method comprising:

• parsing a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• generating a respective representative label for each of a set of identified instances of a data element within said semi-structured data source;
• generating a generalized representative label that matches each of the representative labels generated for the identified instances;
• and
• evaluating a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• storing the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said evaluating comprises evaluating the resilience against a selection criterion by:
• comparing a calculated degree of resilience for the generalized representative label with a calculated degree of resilience of another generalized representative label;
• and
• selecting the generalized representative label having a highest calculated degree of resilience,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to classify data from said plurality of semi-structured data sources.

A program storage device readable by computer, embodying a program of instructions executable by said computer to perform a method to cluster data from a plurality of semi-structured data sources, said method comprising:

• parsing a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• generating a respective representative label for each of a set of identified instances of a data element within said semi-structured data source;
• generating a generalized representative label that matches each of the representative labels generated for the identified instances;
• and
• evaluating a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• storing the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said evaluating comprises evaluating the resilience against a selection criterion by:
• comparing a calculated degree of resilience for the generalized representative label with a calculated degree of resilience of another generalized representative label;
• and
• selecting the generalized representative label having a highest calculated degree of resilience,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to cluster data from said plurality of semi-structured data sources.

A data processing apparatus configured to extract data from a plurality of semi-structured data sources, the apparatus comprising:

• a data processing unit;
• a data storage unit operatively connected to said data processing unit;
• a marker program comprising computer-executable instructions configured to parse a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• a label generator adapted configured to generate a respective representative label for each of a set of identified instances of a data element within a semi-structured data source, and to generate a generalized representative label that matches each of the representative labels generated for the identified instances;
• an evaluator configured to evaluate a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• a storage device configured to store the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said evaluator is further configured to evaluate the resilience against a selection criterion by:
• comparing a calculated degree of resilience for the generalized representative label with a calculated degree of resilience of another generalized representative label;
• and
• selecting the generalized representative label having a highest calculated degree of resilience,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to extract data from said plurality of semi-structured data sources.

A data processing apparatus configured to classify data from a plurality of semi-structured data sources, the apparatus comprising:

• a data processing unit;
• a data storage unit operatively connected to said data processing unit;
• a marker program comprising computer-executable instructions configured to parse a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• a label generator adapted configured to generate a respective representative label for each of a set of identified instances of a data element within a semi-structured data source, and to generate a generalized representative label that matches each of the representative labels generated for the identified instances;
• and
• an evaluator adapted configured to evaluate a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• a storage device configured to store the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said evaluator is further configured to evaluate the resilience against a selection criterion by:
• comparing a calculated degree of resilience for the generalized representative label with a calculated degree of resilience of another generalized representative label;
• and
• selecting the generalized representative label having a highest calculated degree of resilience,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to classify data from said plurality of semi-structured data sources.

A data processing apparatus configured to cluster data from a plurality of semi-structured data sources, the apparatus comprising:

• a data processing unit;
• a data storage unit operatively connected to said data processing unit;
• a marker program comprising computer-executable instructions configured to parse a semi-structured data source to produce a well-formed document that is mapped to a Document Object Model (DOM) tree;
• a label generator configured to generate a respective representative label for each of a set of identified instances of a data element within a semi-structured data source, and to generate a generalized representative label that matches each of the representative labels generated for the identified instances;
• and
• an evaluator configured to evaluate a resilience of the generalized representative label to structural changes of the semi-structured data source by estimating, for a set of possible structural changes to the semi-structured data source, a probability of the generalized representative label satisfying a set of resilience criteria;
• and
• a storage device configured to store the evaluated resilience of the generalized representative label,
• wherein the resilience of a label is evaluated with respect to a known change model that is learned from a history of previous changes to said semi-structured data source, wherein said evaluator is further configured to evaluate the resilience against a selection criterion by:
• comparing a calculated degree of resilience for the generalized representative label with a calculated degree of resilience of another generalized representative label;
• and
• selecting the generalized representative label having a highest calculated degree of resilience,
• wherein said estimating occurs by determining a probability that said respective representative label will match nodes of interest in said DOM tree and determining a probability that said respective representative label will match nodes not of interest in said DOM tree, and
• wherein said resilience of the generalized representative label is used as a measure to cluster data from said plurality of semi-structured data sources.