# US 9,529,863 B1 · Normalizing ingested data sets based on fuzzy comparisons to known data sets > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,529,863 B1 - Document ID: US9529863B1 - Status: Granted - Kind: B1 - Application number: 14/977,368 - Filed: 2015-12-21 - Granted: 2016-12-27 - Earliest claimed: 2015-12-21 - Assignees: Apptio, Inc. - Applicants: Apptio, Inc. - Inventors: Jay Randolf Gindin; Paul Damien McLachlan; Svilen Mihaylov - Web page: https://patentgrep.com/patent/US9529863B1 ## CPC - G06Q 40/00: Finance; Insurance; Tax strategies; Processing of corporate or income taxes - G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Embodiments are directed towards normalizing ingested data sets based on fuzzy comparisons to known data sets. Raw data sets that each include raw records may be provided to an ingestion engine. Ingestion rules and known data sets may be provided based on the raw records. The ingestion engine may be employed to iteratively execute the ingestion rules. A comparison of the raw records to the known data sets may be performed. Contents of the raw records may be transformed into model record values and stored in model records. A score value that indicates a confidence level that the model records are correct may be provided. An association of the one or more ingestion rules used to transform the raw record contents into the model record values for each of the one or more model records may be added to a data model. ## Claims ### Claim 1 A method for ingesting data for a data model using a network computer that employs one or more processors to execute instructions that perform actions, comprising: providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records; providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records; employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including: providing a comparison of one or more portions of the one or more raw records to the one or more known data sets; transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets; storing the one or more model record values in one or more model records; providing a score value that indicates a confidence level that the one or more model records are correct based on the one or more confidence scores; and storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records; and when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including: providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and storing the one or more model records in a data store, wherein the one or more model records are added to the data model. ### Claim 2 (depends on 1) The method of claim 1, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets. ### Claim 3 (depends on 1) The method of claim 1, wherein the one or more known data sets are provided from one or more external sources. ### Claim 4 (depends on 1) The method of claim 1, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison. ### Claim 5 (depends on 1) The method of claim 1, further comprising: when two or more of the one or more ingestion rules matches a raw record, performing further actions: providing at least one model record for each of the matched two or more ingestion rules; and allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules. ### Claim 6 (depends on 1) The method of claim 1, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises: storing a historical record of the modifications made to the one or more model records; and generating a report that includes a source and a type of one or more errors that triggered the interactive modification session. ### Claim 7 (depends on 1) The method of claim 1, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including: caching at least a portion of the one or more raw data sets when network communication is disabled; and providing the cached at least portion of the one or more raw data sets when network communication is enabled. ### Claim 8 A system for ingesting data for a data model, comprising: a network computer, comprising: a transceiver that communicates over the network; a memory that stores at least instructions; and a processor device that executes instructions that perform actions, including: providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records; providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records; employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including: providing a comparison of one or more portions of the one or more raw records to the one or more known data sets; transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets; storing the one or more model record values in one or more model records; providing a score value that indicates a confidence level that the one or more model records are correct based on the one or more confidence scores; and storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records; when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including: providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and storing the one or more model records in a data store, wherein the one or more model records are added to the data model; and a client computer, comprising: a transceiver that communicates over the network; a memory that stores at least instructions; and a processor device that executes instructions that perform actions, including: providing the user-interface to a user; and providing one or more user interactions to the ingestion engine. ### Claim 9 (depends on 8) The system of claim 8, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets. ### Claim 10 (depends on 8) The system of claim 8, wherein the one or more known data sets are provided from one or more external sources. ### Claim 11 (depends on 8) The system of claim 8, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison. ### Claim 12 (depends on 8) The system of claim 8, wherein the network computer processor device executes instructions that perform actions, further comprising: when two or more of the one or more ingestion rules matches a raw record, performing further actions: providing at least one model record for each of the matched two or more ingestion rules; and allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules. ### Claim 13 (depends on 8) The system of claim 8, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises: storing a historical record of the modifications made to the one or more model records; and generating a report that includes a source and a type of one or more errors that triggered the interactive modification session. ### Claim 14 (depends on 8) The system of claim 8, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including: caching at least a portion of the one or more raw data sets when network communication is disabled; and providing the cached at least portion of the one or more raw data sets when network communication is enabled. ### Claim 15 A processor readable non-transitory storage media that includes instructions for ingesting data for a data model, wherein execution of the instructions by a hardware processor performs actions, comprising: providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records; providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records; employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including: providing a comparison of one or more portions of the one or more raw records to the one or more known data sets; transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets; storing the one or more model record values in one or more model records; providing a score value that indicates a confidence level that the one or more model records are correct based on the one or more confidence scores; and storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records; when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including: providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and storing the one or more model records in a data store, wherein the one or more model records are added to the data model. ### Claim 16 (depends on 15) The media of claim 15, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets. ### Claim 17 (depends on 15) The media of claim 15, wherein the one or more known data sets are provided from one or more external sources. ### Claim 18 (depends on 15) The media of claim 15, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison. ### Claim 19 (depends on 15) The media of claim 15, further comprising: when two or more of the one or more ingestion rules matches a raw record, performing further actions: providing at least one model record for each of the matched two or more ingestion rules; and allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules. ### Claim 20 (depends on 15) The media of claim 15, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises: storing a historical record of the modifications made to the one or more model records; and generating a report that includes a source and a type of one or more errors that triggered the interactive modification session. ### Claim 21 (depends on 15) The media of claim 15, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including: caching at least a portion of the one or more raw data sets when network communication is disabled; and providing the cached at least portion of the one or more raw data sets when network communication is enabled. ### Claim 22 A network computer for ingesting data for a data model, comprising: a transceiver that communicates over the network; a memory that stores at least instructions; and a processor device that executes instructions that perform actions, including: providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records; providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records; employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including: providing a comparison of one or more portions of the one or more raw records to the one or more known data sets; transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets; storing the one or more model record values in one or more model records; providing a score value that indicates a confidence level that the one or more model records are correct based at least on the one or more confidence scores; storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records; when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including: providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and storing the one or more model records in a data store, wherein the one or more model records are added to the data model. ### Claim 23 (depends on 22) The network computer of claim 22, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets. ### Claim 24 (depends on 22) The network computer of claim 22, wherein the one or more known data sets are provided from one or more external sources. ### Claim 25 (depends on 22) The network computer of claim 22, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison. ### Claim 26 (depends on 22) The network computer of claim 22, further comprising: when two or more of the one or more ingestion rules matches a raw record, performing further actions: providing at least one model record for each of the matched two or more ingestion rules; and allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules. ### Claim 27 (depends on 22) The network computer of claim 22, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises: storing a historical record of the modifications made to the one or more model records; and generating a report that includes a source and a type of one or more errors that triggered the interactive modification session. ### Claim 28 (depends on 22) The network computer of claim 22, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including: caching at least a portion of the one or more raw data sets when network communication is disabled; and providing the cached at least portion of the one or more raw data sets when network communication is enabled.