Grouping and duplicate removal method in a database

A duplicate removal method comprising the steps of:

• determining whether there is possibility or not for existence of at least another record having a value identical to that of a record by executing a predetermined first process to one or more column values of a plurality of records of a record list included in a database;
• executing a second process for deleting records other than one of the duplicate records having the identical value to a plurality of records other than a plurality of partial records determined to have no possibility for existence of the other records having the same value by said determining step among a plurality of records of said record list;
• and
• a plurality of said partial records and a plurality of said records after said deletion are output as the records after duplicate removal.

A grouping method comprising the steps of:

• generating hash value to each record among a plurality of records of record list included in a data base with a hash function using, as an argument, a value of at least a part of columns of one or more predetermined grouping columns of such records;
• determining whether a value of an effective grouping column and aggregation data are stored or not in one entry corresponding to said hash value among a plurality of entries in a prepared aggregating area;
• calculating the aggregation data when a record is only one record belonging to one group based on data of predetermined aggregating column of said record when said one entry does not store a value of the effective grouping column and the aggregation data;
• storing a value of the grouping column of said record and said calculated aggregation data in said one entry;
• determining whether the values of the stored grouping column is identical to the value of grouping column of each record having generated said hash value when said one entry has already stored the value of the grouping column and aggregation data;
• updating, when the value of said grouping column stored in one entry is identical to the value of grouping column of each record having generated said hash value, said aggregation data stored in said entry to the data obtained when each record having generated said hash value is added to the group corresponding to said one entry based on the data of said aggregation column of each record having generated said hash value;
• storing the data of grouping column of each record and data of aggregation column as the records not grouped when the value of the grouping column stored in said entry is different from the value of the grouping column of each record having generated said hash value;
• outputting, to a plurality of records of said record list, data stored in a plurality of entries of said aggregating area as data in regard to one group after execution of the steps from said generating step to said registering step;
• and
• changing a hash function used in said generating step during the repetition of the steps by repeating the steps from said generating step to said output step to a plurality of records stored as the records not grouped in said storing step until the grouping is made for all records.