Real-time search engine indexing and synchronization system for relational databases

A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:

• receiving a request to index a dataset from a relational database into a search engine index;
• retrieving the dataset from the relational database using a plurality of SQL NTILE queries, each query retrieving a chunk of the dataset by dividing ordered rows into groups;
• converting each chunk to a serialized format compatible with the search engine;
• loading the serialized chunks of the dataset into the search engine index using a bulk application programming interface of the search engine index;
• detecting changes to records in the relational database by using database triggers to capture and store identifiers of changed records in a delta metadata table;
• scanning the delta metadata table to identify the changed records;
• retrieving current versions of the changed records from the relational database using SQL queries stored in a scanner meta table;
• updating corresponding records in the search engine index with the current versions of the changed records;
• and
• responding to search requests using the search engine index.

A device comprising:

• a processor configured to:
• receive a request to index a dataset from a relational database into a search engine index;
• retrieve the dataset from the relational database using a plurality of SQL NTILE queries, each query retrieving a chunk of the dataset by dividing ordered rows into groups;
• converting each chunk to a serialized format compatible with the search engine;
• load the serialized chunks of the dataset into the search engine index using a bulk application programming interface of the search engine index;
• capture changes to records in the relational database by using database triggers to capture and store identifiers of changed records in a delta metadata table;
• periodically scan the delta metadata table to identify the changed records;
• retrieve current versions of the changed records from the relational database using SQL queries stored in a scanner meta table;
• update corresponding records in the search engine index with the current versions of the changed records;
• and
• serve search requests using the search engine index.