US 12,223,278 B2Granted

Automatic data card generation

AssigneeSap SE

InventorHans-Martin Ramsl

Life of this patent

Estimating the term…

Loading patent…

Abstract

Example methods and systems are directed to automatic data card generation for datasets. A data card is a summary that describes quantitative aspects of a dataset, qualitative aspects of a dataset, or both. The data samples and documentation of a dataset are analyzed automatically to determine a number of samples, a primary data type, a license, or any suitable combination thereof. Data formats for data and documentation of the dataset may be automatically recognized. Language of text data may be automatically recognized. The most frequent language for the text data may be identified as the primary language of the dataset. A data card may be created for the dataset. The data card may indicate the number of samples, the data formats used in the data set, the language of text data in the dataset, or any suitable combination thereof.

Claims17

Dependent claims

A method comprising:

analyzing, by one or more processors, a plurality of files of a dataset to determine a primary data type of the dataset;
determining, by the one or more processors, one or more statistical values for the dataset;
determining, by the one or more processors, a count of entries in the dataset;
accessing, by the one or more processors, a data value representing a stored count of the entries;
based on comparing, by the one or more processors, the determined count of the entries with the data value, storing the data value in a data structure;
determining, by the one or more processors, a percentage of the dataset that comprises text in each of a plurality of languages, by performing operations comprising:
splitting each of the plurality of files of the dataset into lines;
for each line, determining a language;
and
determining a percentage of lines in each of the plurality of languages;
based on the determined percentages, determining a primary language of the dataset;
storing, in the data structure, an indication of the primary data type and at least a subset of the one or more statistical values and an indication of the primary language;
determining, by the one or more processors, a first subset of the dataset to be used for training;
determining, by the one or more processors, a second subset of the dataset to be used for validation;
storing, in the data structure, an indication of the first subset and an indication of the second subset;
iteratively generating additional data structures for a set of additional datasets;
receiving, via a user interface, user input to select a set of search criteria, wherein the set
of search criteria comprises the primary data type, the count of entries, and the primary language;
based on the set of search criteria and the data structure, displaying information for the dataset in the user interface, the displayed information including the primary data type, the count of entries, and the primary language;
and
based on a selection of the dataset by the user, training a neural network using the first subset of the dataset.

18 more elements

A system comprising:

a memory that stores instructions;
and
one or more processors configured by the instructions to perform operations comprising:
analyzing a plurality of files of a dataset to determine a primary data type of the dataset;
determining one or more statistical values for the dataset;
determining a count of entries in the dataset;
accessing a data value representing a stored count of the entries:
based on comparing the determined count of the entries with the data value, storing the data value in a data structure;
determining a percentage of the dataset that comprises text in each of a plurality of languages, by performing operations comprising:
splitting each of the plurality of files of the dataset into lines;
for each line, determining a language;
and
determining a percentage of lines in each of the plurality of languages;
based on the determined percentages, determining a primary language of the dataset;
storing, in the data structure, an indication of the primary data type and at least a subset of the one or more statistical values and an indication of the primary language;
determining, by the one or more processors, a first subset of the dataset to be used for training;
determining, by the one or more processors, a second subset of the dataset to be used for validation;
storing, in the data structure, an indication of the first subset and an indication of the second subset;
iteratively generating additional data structures for a set of additional datasets;
receiving, via a user interface, user input to select a set of search criteria, wherein the set of search criteria comprises the primary data type, the count of entries, and the primary language;
based on the set of search criteria and the data structure, displaying information for the dataset in the user interface, the displayed information including the primary data type, the count of entries, and the primary language;
and
based on a selection of the dataset by the user, training a neural network using the first subset of the dataset.

20 more elements

A non-transitory computer-readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

analyzing a plurality of files of a dataset to determine a primary data type of the dataset;
determining one or more statistical values for the dataset;
determining a count of entries in the dataset;
accessing a data value representing a stored count of the entries;
based on comparing the determined count of the entries with the data value, storing the data value in a data structure;
determining a percentage of the dataset that comprises text in each of a plurality of languages, by performing operations comprising:
splitting each of the plurality of files of the dataset into lines;
for each line, determining a language;
and
determining a percentage of lines in each of the plurality of languages;
based on the determined percentages, determining a primary language of the dataset;
storing, in the data structure, an indication of the primary data type and at least a subset of the one or more statistical values and an indication of the primary language;
determining, by the one or more processors, a first subset of the dataset to be used for training;
determining, by the one or more processors, a second subset of the dataset to be used for validation;
storing, in the data structure, an indication of the first subset and an indication of the second subset;
iteratively generating additional data structures for a set of additional datasets;
receiving, via a user interface, user input to select a set of search criteria, wherein the set of search criteria comprises the primary data type, the count of entries, and the primary language;
based on the set of search criteria and the data structure, displaying information for the dataset in the user interface, the displayed information including the primary data type, the count of entries, and the primary language;
and
based on a selection of the dataset by the user, training a neural network using the first subset of the dataset.

17 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/860,912Filed 8 Jul 2022
Published: US 2024/0013004 A111 Jan 2024
Granted: 11 Feb 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06F 16/285
Relational databases: clustering or classification
G06F 40/30
Handling natural language data: semantic analysis
G06F 16/2458
Query processing: special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
G06F 16/31
Of unstructured textual data: indexing; Data structures therefor; Storage structures

Family

US 2024/0013004 A1Pre-grant publication · published 11 Jan 2024
US 12,223,278 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key