Data discovery solution for data curation

A method of generating a set of candidate patterns corresponding a search query, wherein the set of candidate patterns are selected from a plurality of validated patterns that associate a plurality of tags to a plurality of datasets discovered from one or more data sources, and wherein the plurality of validated patterns are generated using a crowdsource-based game application with a plurality of players, the method comprises:

• A) estimating computing resources to be allocated to a job that profiles the one or more data sources based on any of:
• i) information in one or more configuration files indicative of computing resource allocation in the one or more data sources, ii) a count of the one or more data sources, or iii) a record of a previous profiling job, wherein the job uses a scalable parallelization scheme to extract the plurality of datasets, wherein the plurality of datasets represent technical metadata of a plurality of files existing in the one or more data sources;
• B) performing said job for profiling the one or more data sources by:
• dividing said job into a plurality of subtask chunks;
• creating a subtask configuration file for each subtask chunk, wherein the subtask configuration files include:
• information for service connection to each of the one or more data sources;
• parallelization parameters;
• and
• memory usage;
• processing the subtask chunks using a plurality of paralleled workers, wherein each paralleled worker is a continuous delivery pipeline deployed in a container or a virtual machine of the one or more data sources;
• outputting the plurality of datasets from the one or more data sources via the paralleled workers;
• and
• storing the plurality of datasets in a database configured to send the plurality of datasets to the crowdsource-based gaming application for pattern matching and pattern validation;
• C) generating a plurality of unvalidated patterns from the plurality of datasets using a matcher game component of the gaming application by:
• identifying a first player of the plurality of players to transmit a matcher game question, wherein the matcher game question is configured to solicit a player-response that matches a dataset from the plurality of datasets to a tag from the plurality of tags to establish an unvalidated pattern of the plurality of unvalidated patterns;
• transmitting the matcher game question to the first player via a first corresponding gaming application;
• receiving the unvalidated pattern from the first player via the first corresponding gaming application;
• and
• storing the unvalidated pattern to the database;
• D) validating the plurality of unvalidated patterns using a validator game component of the gaming application with the plurality of players by:
• identifying a second player of the plurality of game players to transmit a validator game question to based on a reliability rating of the second player, wherein the reliability rating of the second player is based a previous history of responses to test questions by the second player, and wherein the validator game question is configured to solicit a player-response that confirms or rejects the unvalidated pattern;
• transmitting the validator game question to the second player via a second corresponding gaming application;
• receiving confirmation from the second player regarding the unvalidated pattern via the second corresponding gaming application;
• after receiving a threshold number of confirmations from among the plurality of game players including the second player, establishing a first validated pattern that confirms the unvalidated pattern;
• and
• storing the first validated pattern with the plurality of validated patterns;
• E) receiving the search query from a user via a search engine in a front-end interface;
• and
• F) generating a plurality of candidate patterns corresponding the search query by:
• filtering out a first subset of validated patterns of the plurality of validated patterns that do not match a data type of the search query to establish a set of preliminary candidate patterns;
• reducing a search score of a second subset of validated patterns of the set of preliminary candidate patterns that match a blacklisted pattern associated with the search query by a first amount, wherein the first amount is dynamically determined by a variable user-validation metric;
• increasing the search score of a third subset of validated patterns of the set of preliminary candidate patterns that match a whitelisted pattern associated with the search query by a second amount, wherein the second amount is dynamically determined by the variable user-validation metric;
• increasing the search score of a fourth subset of validated patterns of the set of preliminary candidate patterns that match the search query to any of a plurality of technical metadata of the datasets associated with the fourth subset of validated patterns by a third amount, wherein the third amount is dynamically determined by the variable user-validation metric;
• increasing the search score of a fifth subset of validated patterns of the set of preliminary candidate patterns based on a comparison of the search query to a plurality of normalized field names of the datasets corresponding the validated patterns of the set of preliminary candidate patterns;
• establishing the set of candidate patterns as a subset of the validated patterns of the set of preliminary candidate patterns;
• and
• listing the set of candidate patterns in the front-end interface, wherein the set of candidate patterns include a subset of the validated patterns of the set of preliminary candidate patterns ranked according to corresponding search scores of the validated patterns.

A method of generating a set of candidate patterns corresponding a search query, wherein the set of candidate patterns are selected from a plurality of match patterns that associate a plurality of tags to a plurality of datasets discovered from one or more data sources, the method comprises:

• filtering out a first subset of match patterns of the plurality of match patterns that do not match a data type of the search query to establish a set of preliminary candidate patterns;
• reducing a search score of a second subset of match patterns of the set of preliminary candidate patterns that match a blacklisted pattern associated with the search query by a first amount, wherein the first amount is dynamically determined by a variable user-validation metric;
• increasing the search score of a third subset of match patterns of the set of preliminary candidate patterns that match a whitelisted pattern associated with the search query by a second amount, wherein the second amount is dynamically determined by the variable user-validation metric;
• increasing the search score of a fourth subset of match patterns of the set of preliminary candidate patterns that match the search query to any of a plurality of technical metadata of the datasets associated with the fourth subset of match patterns by a third amount, wherein the third amount is dynamically determined by the variable user-validation metric;
• establishing the set of candidate patterns as a subset of the match patterns of the set of preliminary candidate patterns;
• and
• listing the set of candidate patterns in a user interface, wherein the set of candidate patterns include a subset of the match patterns of the set of preliminary candidate patterns ranked according to corresponding search scores of the match patterns.

A method of validating a plurality of unvalidated patterns using a crowdsource-based game application with a plurality of players, wherein the unvalidated patterns associate a plurality of tags to a plurality of datasets discovered from one or more data sources, the method comprises:

• identifying a first validator game player of the plurality of game players to transmit a validator game question to based on a reliability rating of the first validator game player, wherein the reliability rating of the first validator game player is based on a previous history of responses to test questions by the first validator game player, and wherein the validator game question is configured to solicit a player-response that confirms or rejects an unvalidated pattern of the plurality of unvalidated patterns;
• transmitting the validator game question to the first validator game player via a corresponding gaming application;
• receiving confirmation from the first validator game player that confirms the unvalidated pattern via the corresponding gaming application;
• and
• after receiving a threshold number of confirmations from among the plurality of game players including the first validator game player, establishing a first validated pattern that validates the unvalidated pattern;
• storing the validated pattern with a plurality of validated patterns;
• identifying a second validator game player of the plurality of game players to transmit a test question to based on a reliability rating of the second validator game player, wherein the reliability rating of the second validator game player is based on a previous history of responses to test questions by the second validator game player, wherein the test question is based on a second validated pattern of the the plurality of validated patterns;
• transmitting the test question to the second validator game player via the corresponding gaming application;
• receiving a response to the test question from the second validator game player;
• and
• revising the reliability rating of the second validator game player based on whether the response to the test question matches the second validated pattern.

A system for generating a set of candidate patterns corresponding a search query, wherein the set of candidate patterns are selected from a plurality of match patterns that associate a plurality of tags to a plurality of datasets, and wherein the plurality of match patterns are generated using a crowdsource-based game application with a plurality of players, the system comprises:

• a first memory including a first database configured to store the plurality of match patterns;
• a search engine configured to receive a search query and return the set of candidate patterns;
• ranked according to corresponding search scores of the validated pattern set of candidate patterns a search result including a plurality of match patterns with a search score corresponding each match pattern indicative of a corresponding level of confidence between the match pattern and the search query;
• a crowdsource-based gaming application stored in a second memory, the crowdsource-based gaming application including a matcher game component, wherein the matcher game component causes a first executing processor to:
• identify a matcher game player of the plurality of players to transmit a matcher game question, wherein the matcher game question is configured to solicit a player-response that matches a dataset from the plurality of datasets to a tag from the plurality of tags to establish a match pattern of the plurality of match patterns;
• transmit the matcher game question to the matcher game player via a corresponding gaming application;
• receive the match pattern from the matcher game player via the corresponding gaming application;
• and
• store the match pattern to a second database;
• a recommendation engine stored in a third memory, the recommendation engine including a recommendation algorithm, wherein the recommendation algorithm causes a second executing processor to:
• filter out a first subset of match patterns of the plurality of match patterns that do not match a data type of the search query to establish a set of preliminary candidate patterns;
• reduce a search score of a second subset of match patterns of the set of preliminary candidate patterns that match a blacklisted pattern associated with the search query by a first amount, wherein the first amount is dynamically determined by a variable user-validation metric;
• increase the search score of a third subset of match patterns of the set of preliminary candidate patterns that match a whitelisted pattern associated with the search query by a second amount, wherein the second amount is dynamically determined by the variable user-validation metric;
• increase the search score of a fourth subset of match patterns of the set of preliminary candidate patterns that match the search query to any of a plurality of technical metadata of the datasets associated with the fourth subset of match patterns by a third amount, wherein the third amount is dynamically determined by the variable user-validation metric;
• and
• establish the set of candidate patterns as a subset of the match patterns of the set of preliminary candidate patterns;
• and
• a front-end interface configured to list the match patterns of the set of candidate patterns with the match patterns ranked according to the search score corresponding each match pattern;
• a front-end user interface configured to list the set of candidate patterns in the front-end interface, wherein the set of candidate patterns include a subset of the match patterns of the set of preliminary candidate patterns ranked according to corresponding search scores of the match patterns.