Data transformation to maintain detailed user information in a data warehouse
AssigneeMicrosoft Corporation
InventorsKevin Paul Kornelson, Murali Vajjiravel, Rajeev Prasad, Paul D. Clark, Brian Burdick, Tarek Najm
Life of this patent
Estimating the term…
Abstract
Transforming data prior to loading the data into a data warehouse. Software of the invention partitions data records received from a plurality of servers and performs sequential file management operations and identifier management operations on each of the partitions prior to loading the data records into the data warehouse. Data records transformed according to the invention are easier to load into the data warehouse and easier to manipulate after loading. The invention enables analysis and data mining of tens of terabytes of user level detail data and summary data.
Claims22
A method of transforming data in a data collection and warehousing system comprising:
- receiving a plurality of individual log files from a plurality of servers, said log files each comprising a data record and at least one partition key value corresponding thereto;
- partitioning the received data records by assigning each of the data records to one of a plurality of partitions based on the partition key value corresponding to the data record, each of the partitions having one or more of the partition key values associated therewith;
- and
- generating a fact table for each of the partitions, said fact table comprising the partitioned data records and corresponding partition key values, wherein the partition key values comprise one or more of the following:
- a user identifier, a machine identifier, a time interval a region, and a data source.
A method of transforming data in a data collection and warehousing system, said method comprising:
- receiving a plurality of individual log files from a plurality of servers, said log files each comprising a data record and a partition key value corresponding thereto, wherein the partition key values comprise one or more of the following:
- a user identifier, a machine identifier, a time interval, a region, and a data source;
- sorting the received data records according to the corresponding partition key values;
- merging the sorted data records and corresponding partition key values with other data records and other corresponding partition key values, said other data records and other corresponding partition key values being previously received and sorted;
- mapping each of the partition key values to another key value, said other key value representing a unit of information smaller than the partition key value associated with the merged data records;
- and
- generating a dimension table comprising the merged data records and other key values after mapping.
Dependencies are inferred from the claim text.