Automated identification of duplicate information objects

An apparatus for determining whether a particular information object is a duplicate of a separate information object found in a plurality of separate information objects in which the particular information object and each of the plurality of separate information objects comprises a plurality of data fields for storing a plurality of data values associated with a plurality of data types that allows an identical data value in the plurality of data values to be stored in different data fields of the plurality of data fields for the particular information object and the plurality of separate information objects, the apparatus comprising:

• at least one processor;
• and
• at least one memory including stored therein executable computer program code, the at least one memory and the executable computer program code configured to, with execution of the computer program code by the at least one processor,
• cause the apparatus to:
• for each respective separate information object of the plurality of separate information objects, combining, based at least in part on the data type associated with each data value of the plurality of data types, one or more data values stored in the plurality of data fields for the respective separate information object to generate a flattened array data structure for the respective separate information object comprising a plurality of array data fields, wherein each array data field:
• (i) is associated with a unique data type of the plurality of data types and (ii) stores a subset of the plurality of data values that is associated with the unique data type of the array data field;
• combine, based at least in part on the data type associated with the each data value of the plurality of data types, one or more data values stored in the plurality of data fields for the particular information object to generate a flattened array data structure for the particular information object comprising a plurality of array data fields, where each array data field:
• (i) is associated with a unique data type of the plurality of data types and (ii) stores a subset of the plurality of data values that is associated with the unique data type of the array data field;
• and
• for each respective separate information object:
• performing an array intersection identification operation on the flattened array data structure for the particular information object and the flattened array data structure for the respective separate information object to generate, for each data type, a flattened array intersection comprising one or more data values in a corresponding array data field of the flattened array data structure for the particular information object that have identical matches in a corresponding array data field of the flattened array data structure for the respective separate information object;
• determining whether the particular information object is an exact match of the respective separate information object if each flattened array intersection comprises all of the one or more data values of the corresponding array data field for the particular information object;
• determining whether the particular information object is a partial match of the respective separate information object if each flattened array intersection comprises a non-holistic subset of the one or more data values of the corresponding array data field whose ration exceeds a required partial matching degree;
• and
• initiating an automated action based at least in part on whether the particular information object is the exact match of the respective separate information object and whether the particular information object is the partial match of the respective separate information object.

A non-transitory computer-readable storage medium comprising instructions for determining whether a particular information object is a duplicate of a separate information object found in a plurality of separate information objects in which the particular information object and each of the plurality of separate information objects comprises a plurality of data fields for storing a plurality of data values associated with a plurality of data types that allows an identical data value in the plurality of data values to be stored in different data fields of the plurality of data fields for the particular information object and the plurality of separate information objects, the instructions being configured to be executed by one or more processors to at least perform operations configured to:

• for each respective separate information object of the plurality of separate information objects, combining, based at least in part on the data type associated with each data value of the plurality of data types, one or more data values stored in the plurality of data fields for the respective separate information object to generate a flattened array data structure for the respective separate information object comprising a plurality of array data fields, wherein each array data field;
• (i) is associated with a unique data type of the plurality of data types and (ii) stores a subset of the plurality of data values that is associated with the unique data type of the array data field;
• combine, based at least in part on the data type associated with each value of the plurality of data types, one or more data values stored in the plurality of data fields for the particular information object to generate a flattened array data structure for the particular information object comprising a plurality of array data fields, wherein each array data field:
• (i) is associated with a unique data, type of the plurality of data types and (ii) stores a subset of the plurality of data values that is associated with the unique data type of the array data field;
• and
• for each respective separate information object;
• performing an array intersection identification operation on the flattened array data structure for the particular information object and the flattened array data structure for the respective separate information object to generate, for each data type, a flattened array intersection comprising one or more data values in a corresponding array data field of the flattened array data structure for the particular information object that have identical matches in a corresponding array data field of the flattened array data structure for the respective separate information object;
• determining whether the particular information object is an exact match of the respective separate information object each flattened array intersection comprises all of the one or more data values of the corresponding array data field for the particular information object;
• determining whether the particular information object is a partial match of the respective separate information object if each flattened array intersection comprises a non-holistic Subset of the one or more data values of the corresponding array data field whose ratio exceeds a required partial matching degree;
• and
• initiating an automated action based at least in part on whether the particular information object is the exact match of the respective separate information object and whether the particular information object is the partial match of the respective separate information object.