Method and system for maintaining consistency during multi-threaded processing of LDIF data

A method for loading data into a directory, the method comprising:

• obtaining LDIF (Lightweight Directory Access Protocol (LDAP) Data Interchange Format) entries;
• associating a priority value with each LDIF entry,
• wherein the priority value is determined based on a number of delimiters in the LDIF entry, and
• wherein the entries having a higher number of delimiters are assigned a lower priority and entries having a lower number of delimiters are assigned a higher priority value;
• adding the LDIF entries into an LDAP directory in accordance with their associated priority values using multiple loading threads,
• wherein the LDIF entries are added to the LDAP directory in an order of highest indicated priority to lowest indicated priority,
• wherein the LDIF entries that are associated with priority values indicating equal priority being a subset of LDIF entries,
• wherein the LDIF entries from the subset of entries may be added in any order,
• wherein the priority value for an LDIF entry is based on the number of delimiters in a distinguished name within an LDIF entry,
• wherein multiple reader threads obtain the entries from a data source, and
• wherein the LDIF entries are not presented in priority order while the LDIF entries are being obtained;
• obtaining the LDIF entries in a consecutive manner without performing an initial pass through the data source;
• storing the LDIF entries and their associated priority values into a data structure;
• retrieving the LDIF entries from the data structure prior to adding the LDIF entries to the LDAP directory,
• wherein the LDIF entries are retrieved in an order based on the priority values associated with the LDIF entries,
• wherein LDIF entries are retrieved from the data structure in an order of highest indicated priority to lowest indicated priority,
• wherein the LDIF entries from the subset of LDIF entries may be retrieved in any order;
• employing a single reader thread from the multiple reader threads to read LDIF entries from a data source and put the LDIF entries into the data structure;
• employing the multiple loading threads to get LDIF entries from the data structure and add the LDIF entries to the LDAP directory;
• managing the data structure using a data structure manager, the single reader thread and the multiple loading threads interfacing with the data structure manager to put and get LDIF entries to and from the data structure;
• switching between a getting mode of operation in which the reader thread is executing and a putting mode of operation in which the multiple loading threads are executing and a putting mode of operation in which the multiple loading threads are executing so that either the reader thread is executing or the multiple loading threads are executing.