# US 2019/0155822 A1 · System For Continuous Monitoring Of Data Quality In A Dynamic Feed Environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0155822 A1 - Document ID: US20190155822A1 - Status: Published application - Kind: A1 - Application number: 16/257,936 - Filed: 2019-01-25 - Published: 2019-05-23 - Earliest claimed: 2015-07-30 - Assignees: AT&T Intellectual Property I, L.P. - Applicants: AT&T Intellectual Property I, L.P. - Inventors: Vladislav Shkapenyuk; Tamraparni Dasu; Divesh Srivastava; Deborah Swayne - Web page: https://patentgrep.com/patent/US20190155822A1 ## CPC - G06F 11/3082: {the data filtering being achieved by aggregating or compressing the monitored data} - G06F 16/283: {Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP} - G06F 16/285: {Clustering or classification} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system for providing continuous monitoring of data quality in a dynamic feed environment is disclosed. In particular, the system utilizes a feed inspection tool to detect anomalies in data gathering detected from feed metadata and anomalies in data measurement detected based on file contents. In order to do so, the feed inspection tool may aggregate, for a plurality of aggregation intervals, data feeds and associated metadata feeds. Once the data feeds and metadata feeds are aggregated, the feed inspection tool may generate, for a baseline model feed, baseline statistical models by utilizing historical data of the aggregated feeds in sliding windows of different lengths. The feed inspection tool may then identify, for a plurality of monitoring time delays, data outliers by comparing the aggregated feeds with the baseline model feed. A data quality feed based on the data outliers identified may then be generated and published. ## Claims ### Claim 1 A system, comprising: a memory that stores instructions; and a processor that executes the instructions to perform operations, the operations comprising: passively monitoring, by utilizing a feed inspection tool of a computing device, data feeds and metadata feeds associated with the data feeds; aggregating, for a plurality of aggregation intervals, the data feeds and the metadata feeds by combining the data feeds and the metadata feeds into an aggregation feed; generating, for a baseline model feed and by utilizing model parameters provided by the feed inspection tool, baseline models at multiple scales, wherein the baseline models are generated by utilizing the feed inspection tool of the computing device; identifying, for a plurality of monitoring time delays, data outliers by comparing the aggregation feed for a current aggregation interval to the baseline model feed; and generating a data quality feed based on the data outliers identified, wherein a configurable subset of entire files are sampled by utilizing a longitudinal sample that is selected when a full feed rate for delivering the files is greater than a number of files a subscriber of the data feeds and the data quality feed is capable of handling, wherein a set of random files across the data feeds are also sampled when generating the data quality feed so as to capture correlations among the data feeds that are not capturable using the longitudinal sample. ### Claim 2 (depends on 1) The system of claim 1, wherein the operations further comprise deriving group-by variables and quantitative variables from the data feeds and the metadata feeds. ### Claim 3 (depends on 2) The system of claim 2, wherein the operations further comprise computing a statistical summary for the quantitative variables for each combination of the group-by variables. ### Claim 4 (depends on 1) The system of claim 1, wherein the operations further comprise receiving the data feeds and the metadata feeds from a data feed management system. ### Claim 5 (depends on 1) The system of claim 1, wherein the operation of generating the data quality feed further comprises generating the data quality feed based on sampling the configurable subset of entire files in the data feeds instead of sampling all the files in the data feeds. ### Claim 6 (depends on 1) The system of claim 1, wherein the operations further comprise publishing the data quality feed so that the data quality feed is accessible to the subscriber of the data quality feed. ### Claim 7 (depends on 1) The system of claim 1, wherein the operations further comprise subscribing to the data feeds and the metadata feeds. ### Claim 8 (depends on 1) The system of claim 1, wherein the generating of the baseline models at the multiple scales further comprises generating the baseline models by utilizing historical data associated with the aggregation feed in sliding windows of different lengths. ### Claim 9 (depends on 1) The system of claim 1, wherein the operations further comprise facilitating access to the data quality feed for additional subscribers. ### Claim 10 (depends on 1) The system of claim 1, wherein the operations further comprise generating a report or graph that visually identifies the data outliers. ### Claim 11 (depends on 1) The system of claim 1, wherein the operations further comprise receiving a request to access a logical alert feed of the data quality feed, wherein the logical alert feed includes alerts computed for a specified time delay. ### Claim 12 (depends on 1) The system of claim 1, wherein the operations further comprise detecting an error in processes associated with the data feeds. ### Claim 13 (depends on 1) The system of claim 1, wherein the operations further comprise generating a time series plot that includes the data outliers. ### Claim 14 A method, comprising: monitoring, by utilizing a feed inspection tool of a computing device, data feeds and metadata feeds associated with the data feeds; aggregating, for a plurality of aggregation intervals, the data feeds and the metadata feeds by combining the data feeds and the metadata feeds into an aggregation feed; generating, for a baseline model feed and by utilizing model parameters provided by the feed inspection tool, baseline models at multiple scales, wherein the baseline models are generated by utilizing the feed inspection tool of the computing device; determining, for a plurality of monitoring time delays, data outliers by comparing the aggregation feed for a current aggregation interval to the baseline model feed; and creating, by utilizing instructions from a memory that are executed by a processor, a data quality feed based on the data outliers identified, wherein a configurable subset of entire files are sampled by utilizing a longitudinal sample that is selected when a full feed rate for delivering the files is greater than a number of files a subscriber of the data feeds and the data quality feed is capable of handling, wherein a set of random files across the data feeds are also sampled when generating the data quality feed so as to capture correlations among the data feeds that are not capturable using the longitudinal sample. ### Claim 15 (depends on 14) The method of claim 14, further comprising detecting a change in a distribution for records in the data feeds by analyzing contents of the data feeds. ### Claim 16 (depends on 14) The method of claim 14, further comprising outputting an alert based on the data outliers. ### Claim 17 (depends on 14) The method of claim 14, further comprising generating a report or graph that visually identifies the data outliers. ### Claim 18 (depends on 14) The method of claim 14, further comprising subscribing to an unmatched data feed, wherein the unmatched data feed does not match any logical feeds defined by the subscriber. ### Claim 19 (depends on 18) The method of claim 18, further comprising labeling unmatched files of the unmatched feed to a data feed of the data feeds that shares a similarity to the unmatched files. ### Claim 20 A computer-readable device comprising instructions, which when executed by a processor, cause the processor to perform operations comprising: analyzing, by utilizing a feed inspection tool of a computing device, data feeds and metadata feeds associated with the data feeds; aggregating, for a plurality of aggregation intervals, the data feeds and the metadata feeds by combining the data feeds and the metadata feeds into an aggregation feed; generating, for a baseline model feed and by utilizing model parameters provided by the feed inspection tool, baseline models at multiple scales, wherein the baseline models are generated by utilizing the feed inspection tool of the computing device; determining, for a plurality of monitoring time delays, data outliers by comparing the aggregation feed for a current aggregation interval to the baseline model feed; and generating, by utilizing instructions from a memory that are executed by a processor, a data quality feed based on the data outliers identified, wherein a configurable subset of entire files are sampled by utilizing a longitudinal sample that is selected when a full feed rate for delivering the files is greater than a number of files a subscriber of the data feeds and the data quality feed is capable of handling, wherein a set of random files across the data feeds are also sampled when generating the data quality feed so as to capture correlations among the data feeds that are not capturable using the longitudinal sample.