Back to results
US 12,517,952 B2Granted
Semi-structured data decomposition
AssigneeInternational Business Machines Corporation
InventorsLei Tian, Han Zhang, Yi Shao, Ying Xu, Fu Ju An, Yi Li
Life of this patent
Estimating the term…
Abstract
A computer-implemented technique for decomposing semi-structured data is provided. In this technique, metadata for a predetermined number of records can be collected from semi-structured data that includes several records. A structured format is generated based on the metadata and the plurality of records is decomposed with the structured format.
Claims10
A computer implemented method, comprising:
- determining, by one or more processing units, a predetermined number of different records from semi-structured data from which to collect metadata therefrom, wherein the semi-structured data comprises a plurality of records respectively comprising one or more attributes represented by a pair of attribute name and attribute value and wherein the predetermined number is smaller than a quantity of the plurality of records;
- collecting, by one or more processing units, metadata for the predetermined number of different records from the semi-structured data by collecting, for each of the predetermined number of different records, at least a respective attribute name and at least one nested attribute associated with the respective attribute name, the at least one nested attribute comprising a child attribute name and a child attribute value;
- generating, by the one or more processing units, a structured format based on the metadata, by determining a first presence ratio of each respective attribute name in relation to the predetermined number of different records and, in response to a respective associated presence ratio being higher than a ratio threshold, adding the respective attribute name as a first component of the structured format and determining a second presence ratio of a child attribute name that is associated with the first component in relation to the predetermined number of different records and, in response to the second presence ratio being higher than a ratio threshold for the child attribute name, adding the child attribute name as a second component of the structured format and removing the first component from the structured format;
- and
- decomposing, by the one or more processing units, the plurality of records according to the structured format by extracting, for the plurality of records, an attribute value associated with the second component of the structured format, thereby generating structured data from the semi-structured data in accordance with the structured format.
2 more elements
An apparatus, comprising:
- one or more processors;
- a memory coupled to at least one of the one or more processors;
- and
- computer program instructions stored in the memory and executed by at least one of the one or more processors to:
- determine a predetermined number of different records from semi-structured data from which to collect metadata therefrom, wherein the semi-structured data comprises a plurality of records respectively comprising one or more attributes represented by a pair of attribute name and attribute value and wherein the predetermined number of different records is smaller than a quantity of the plurality of records;
- collect metadata for the predetermined number of different records from the semi-structured data by collecting at least a respective attribute name and by collecting at least one nested attribute associated with the respective attribute name, the at least one nested attribute comprising a child attribute name and a child attribute value for each of the predetermined number of different records;
- generate a structured format based on the metadata by determining a first presence ratio of each respective attribute name in relation to the predetermined number of different records and, in response to a respective associated presence ratio being higher than a ratio threshold, adding the respective attribute name as a first component of the structured format and determining a second presence ratio of a child attribute name that is associated with the first component in relation to the predetermined number of different records and, in response to the second presence ratio being higher than a ratio threshold for the child attribute name, adding the child attribute name as a second component of the structured format and removing the first component from the structured format;
- and
- decompose the plurality of records according to the structured format by extracting, for the plurality of records, an attribute value associated with the second component of the structured format, thereby generating structured data from the semi-structured data in accordance with the structured format.
6 more elements
A computer program product, comprising a computer readable storage medium having computer program instructions embodied therewith, the computer program instructions executable by a one or more processors to cause the one or more processors to:
- determine a predetermined number of different records from semi-structured data from which to collect metadata therefrom, wherein the semi-structured data comprises a plurality of records respectively comprising one or more attributes represented by a pair of attribute name and attribute value and wherein the predetermined number of different records is smaller than a quantity of the plurality of records;
- collect metadata for the predetermined number of different records from the semi-structured data by collecting at least a respective attribute name and by collecting at least one nested attribute associated with the respective attribute name, the at least one nested attribute comprising a child attribute name and a child attribute value for each of the predetermined number of different records;
- generate a structured format based on the metadata by determining a first presence ratio of each respective attribute name in relation to the predetermined number of different records and, in response to a respective associated presence ratio being higher than a ratio threshold, adding the respective attribute name as a first component of the structured format and determining a second presence ratio of a child attribute name that is associated with the first component in relation to the predetermined number of different records and, in response to the second presence ratio being higher than a ratio threshold for the child attribute name, adding the child attribute name as a second component of the structured format and removing the first component from the structured format;
- and
- decompose the plurality of records according to the structured format by extracting, for the plurality of records, an attribute value associated with the second component of the structured format, thereby generating structured data from the semi-structured data in accordance with the structured format.
2 more elements
Dependencies are inferred from the claim text.