US 11,537,905 B2 · Inference-based assignment of data type to data

Abstract

Embodiments associate a relevant semantic data type (e.g., date) with incoming raw data (e.g., a column of digits) which lacks metadata. Assignment of semantic data type is inferred from a plurality of features. A first step determines a first feature comprising success rate in converting the raw data into various semantic data types. Then, alignment between observed/reference distributions of other features (e.g., data first digit, data length) is determined per-semantic data type. Total scores for each semantic data type are calculated from the combined features, and used as a basis for ranking the semantic data types. The total scores may reflect a weighting of the various features. In a second step, top-ranked semantic data types may be further differentiated from one another by applying additional features. User feedback regarding accuracy of semantic data type assignment, may be incorporated into training data used to modify the feature reference distributions.

Claims20

Dependent claims

A computer-implemented method comprising:

receiving a plurality of data points lacking associated metadata;
determining a first score of a first feature comprising a success rate converting the plurality of data points to a semantic data type;
determining a second score from alignment between an observed distribution of a second feature in the plurality of data points, and a reference distribution of the second feature in the semantic data type;
calculating a total score for the semantic data type from the first and second scores;
determining an initial ranking of the total score in comparison with another total score reflecting the first and second features in another semantic data type;
identifying a third feature differentiating between the semantic data type and the other semantic data type;
assigning a final ranking to the total score and the other total score by,
determining alignment between an observed distribution of the third feature in the plurality of data points, and reference distributions of the third feature in the semantic data type and in the other semantic data type;
assigning the semantic data type to the plurality of data points based upon the final ranking;
and
storing the plurality of data points with metadata indicating the semantic data type.

8 more elements

A non-transitory computer readable storage medium embodying a computer program for performing a method, said method comprising:

receiving a plurality of data points lacking associated metadata;
determining a first score of a first feature comprising a success rate converting the plurality of data points to a semantic data type;
determining a second score from alignment between an observed distribution of a second feature in the plurality of data points, and a reference distribution of the second feature in the semantic data type;
calculating a total score for the semantic data type from the first and second scores;
determining an initial ranking of the total score in comparison with another total score reflecting the first and second features in another semantic data type;
identifying a third feature differentiating between the semantic data type and the other semantic data type;
assigning a final ranking to the total score and the other total score by,
determining alignment between an observed distribution of the third feature in the plurality of data points, and reference distributions of the third feature in the semantic data type and in the other semantic data type;
assigning the semantic data type to the plurality of data points based upon the final ranking;
and
storing the plurality of data points with metadata indicating the semantic data type.

8 more elements

A computer system comprising:

one or more processors;
a software program, executable on said computer system, the software program configured to cause an in-memory database engine to:
receive a plurality of data points lacking associated metadata;
determine a first score of a first feature comprising a success rate converting the plurality of data points to a semantic data type;
determine a second score from alignment between an observed distribution of a second feature in the plurality of data points, and a reference distribution of the second feature in the semantic data type;
calculate a total score for the semantic data type from the first and second scores;
determine an initial ranking of the total score in comparison with another total score reflecting the first and second features in another semantic data type;
identify a third feature differentiating between the semantic data type and the other semantic data type;
assign a final ranking to the total score and the other total score by,
determining alignment between an observed distribution of the third feature in the plurality of data points, and reference distributions of the third feature in the semantic data type and in the other semantic data type;
assign the semantic data type to the plurality of data points based upon the final ranking;
and
store the plurality of data points in a column of a table of an in-memory database with metadata indicating the semantic data type.

10 more elements

Dependencies are inferred from the claim text.

Inference-based assignment of data type to data

Life of this patent

Inference-based assignment of data type to data

Life of this patent

Abstract

Claims20