# US 2020/0394166 A1 · Rules-based dataset cleaning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0394166 A1 - Document ID: US20200394166A1 - Status: Published application - Kind: A1 - Application number: 16/449,207 - Filed: 2019-06-21 - Published: 2020-12-17 - Earliest claimed: 2019-06-13 - Assignees: Palantir Technologies Inc. - Applicants: Palantir Technologies, Inc. - Inventors: Shelby Vanhooser - Web page: https://patentgrep.com/patent/US20200394166A1 ## CPC - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F 16/221: {Column-oriented storage; Management thereof} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract One or more processors store rules for performing rules-based cleaning operations on a plurality of datasets, wherein each rule comprises one or more functions to be executed against a dataset during the rules-based cleaning operations, the one or more functions each having one or more associated conditions and actions, wherein the one or more actions are performed on the dataset responsive to the one or more associated conditions being satisfied. The one or more processors further apply the rules to each of the plurality of datasets to perform the rules-based cleaning operations. To apply the rules to a given dataset, the one or more processors identify an ordered list of the one or more functions to be executed with respect to the given dataset during the rules-based cleaning operations and determine, for each of the one or more functions, whether the given dataset satisfies one or more conditions associated with a respective function of the one or more functions. Responsive to the given dataset satisfying the one or more conditions associated with the respective function, the one or more processors perform, on the given dataset, one or more actions associated with the respective function and provide a derived dataset comprising at least one modification to the given dataset resulting from the one or more actions associated with the respective function. ## Claims ### Claim 1 A method comprising: storing rules for performing rules-based cleaning operations on a plurality of datasets, wherein each rule comprises one or more functions to be executed against a dataset during the rules-based cleaning operations, the one or more functions each having one or more associated conditions and actions, wherein the one or more actions are performed on the dataset responsive to the one or more associated conditions being satisfied; and applying the rules to each of the plurality of datasets to perform the rules-based cleaning operations, wherein applying the rules to a given dataset comprises: identifying an ordered list of the one or more functions to be executed with respect to the given dataset during the rules-based cleaning operations; determining, for each of the one or more functions, whether the given dataset satisfies one or more conditions associated with a respective function of the one or more functions; responsive to the given dataset satisfying the one or more conditions associated with the respective function, performing, on the given dataset, one or more actions associated with the respective function; and providing a derived dataset comprising at least one modification to the given dataset resulting from the one or more actions associated with the respective function. ### Claim 2 (depends on 1) The method of claim 1, wherein each of the plurality of datasets comprises one or more columns each having corresponding column metadata, and wherein performing the one or more actions comprises modifying at least one column metadata of at least one of the one or more columns from the dataset. ### Claim 3 (depends on 2) The method of claim 2, wherein each column metadata comprises at least one of a column name or a column data type, and wherein performing the one or more actions comprises modifying at least one of a column name or a column data type of at least one of the one or more columns from the first dataset. ### Claim 4 (depends on 1) The method of claim 1, wherein determining whether the given dataset satisfies the one or more conditions associated with the respective function comprises determining whether at least one of a name, identifier, path, or storage location associated with the given dataset matches a defined parameter. ### Claim 5 (depends on 1) The method of claim 1, wherein the rules are applied to each of the plurality of datasets automatically responsive to a triggering event, the triggering event comprising at least one of a periodic interval having elapsed since a previous rules-based cleaning operation was performed, an addition of a new dataset to the plurality of datasets, or receipt of a request to perform a rules-based cleaning operation. ### Claim 6 (depends on 1) The method of claim 1, wherein applying the rules to a given dataset further comprises: determining whether the derived dataset satisfies one or more conditions associated with a subsequent function in the ordered list of the one or more functions; and responsive to the derived dataset satisfying the one or more conditions associated with the subsequent function, performing, on the derived dataset, one or more actions associated with the subsequent function. ### Claim 7 (depends on 1) The method of claim 1, further comprising: generating an indication of the functions for which the one or more associated conditions were satisfied and the associated one or more actions were performed; and associating the indication with the derived dataset. ### Claim 8 (depends on 1) The method of claim 1, wherein the rules are stored as an object model, the object model comprising one or more objects that each represent an action of the one or more actions to be performed with respect to the plurality of datasets, one or more objects that each represent a function of the one or more functions that invokes the one or more actions, and one or more objects that each represent a condition of the one or more conditions defining whether to perform the one or more actions. ### Claim 9 (depends on 8) The method of claim 8, wherein the object model further comprises one or more objects that each represent a sequencer defining the ordered list of the one or more functions to be executed, and an object representing a cleaner that orchestrates the rules-based cleaning operations. ### Claim 10 A system comprising: memory; and one or more processors coupled to the memory, the one or more processors to: store rules for performing rules-based cleaning operations on a plurality of datasets, wherein each rule comprises one or more functions to be executed against a dataset during the rules-based cleaning operations, the one or more functions each having one or more associated conditions and actions, wherein the one or more actions are performed on the dataset responsive to the one or more associated conditions being satisfied; and apply the rules to each of the plurality of datasets to perform the rules-based cleaning operations, wherein to apply the rules to a given dataset, the one or more processors to: identify an ordered list of the one or more functions to be executed with respect to the given dataset during the rules-based cleaning operations; determine, for each of the one or more functions, whether the given dataset satisfies one or more conditions associated with a respective function of the one or more functions; responsive to the given dataset satisfying the one or more conditions associated with the respective function, perform, on the given dataset, one or more actions associated with the respective function; and provide a derived dataset comprising at least one modification to the given dataset resulting from the one or more actions associated with the respective function. ### Claim 11 (depends on 10) The system of claim 10, wherein each of the plurality of datasets comprises one or more columns each having corresponding column metadata, wherein to perform the one or more actions, the one or more processors to modify at least one column metadata of at least one of the one or more columns from the dataset. ### Claim 12 (depends on 11) The system of claim 11, wherein each column metadata comprises at least one of a column name or a column data type, and wherein to perform the one or more actions, the one or more processors to modify at least one of the column name or the column data type of at least one of the one or more columns from the first dataset. ### Claim 13 (depends on 10) The system of claim 10, wherein to determine whether the given dataset satisfies the one or more conditions associated with the respective function, the one or more processors to determine whether at least one of a name, identifier, path, or storage location associated with the given dataset matches a defined parameter. ### Claim 14 (depends on 10) The system of claim 10, wherein the rules are applied to each of the plurality of datasets automatically responsive to a triggering event, the triggering event comprising at least one of a periodic interval having elapsed since a previous rules-based cleaning operation was performed, an addition of a new dataset to the plurality of datasets, or receipt of a request to perform a rules-based cleaning operation. ### Claim 15 (depends on 10) The system of claim 10, wherein to apply the rules to a given dataset, the one or more processors further to: determine whether the derived dataset satisfies one or more conditions associated with a subsequent function in the ordered list of the one or more functions; and responsive to the derived dataset satisfying the one or more conditions associated with the subsequent function, perform, on the derived dataset, one or more actions associated with the subsequent function. ### Claim 16 A non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: store rules for performing rules-based cleaning operations on a plurality of datasets, wherein each rule comprises one or more functions to be executed against a dataset during the rules-based cleaning operations, the one or more functions each having one or more associated conditions and actions, wherein the one or more actions are performed on the dataset responsive to the one or more associated conditions being satisfied; and apply the rules to each of the plurality of datasets to perform the rules-based cleaning operations, wherein to apply the rules to a given dataset, the instructions cause the one or more processors to: identify an ordered list of the one or more functions to be executed with respect to the given dataset during the rules-based cleaning operations; determine, for each of the one or more functions, whether the given dataset satisfies one or more conditions associated with a respective function of the one or more functions; responsive to the given dataset satisfying the one or more conditions associated with the respective function, perform, on the given dataset, one or more actions associated with the respective function; and provide a derived dataset comprising at least one modification to the given dataset resulting from the one or more actions associated with the respective function. ### Claim 17 (depends on 16) The non-transitory computer readable storage medium of claim 16, wherein each of the plurality of datasets comprises one or more columns each having corresponding column metadata, wherein to perform the one or more actions, the instructions cause the one or more processors to modify at least one column metadata of at least one of the one or more columns from the dataset. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein each column metadata comprises at least one of a column name or a column data type, and wherein to perform the one or more actions, the instructions cause the one or more processors to modify at least one of the column name or the column data type of at least one of the one or more columns from the first dataset. ### Claim 19 (depends on 16) The non-transitory computer readable storage medium of claim 16, wherein to determine whether the given dataset satisfies the one or more conditions associated with the respective function, the instructions cause the one or more processors to determine whether at least one of a name, identifier, path, or storage location associated with the given dataset matches a defined parameter. ### Claim 20 (depends on 16) The non-transitory computer readable storage medium of claim 16, wherein the rules are applied to each of the plurality of datasets automatically responsive to a triggering event, the triggering event comprising at least one of a periodic interval having elapsed since a previous rules-based cleaning operation was performed, an addition of a new dataset to the plurality of datasets, or receipt of a request to perform a rules-based cleaning operation.