Systems and methods for indexing source code in a search engine

A system for transmitting content from a source code management (SCM) version of a repository maintained by a source code management system to a separate search engine (SE) version of the repository maintained by a search engine system, the system comprising:

• a processor,
• a communication interface, and
• a non-transitory computer-readable storage medium storing sequences of instructions, which when executed by the processor, cause the processor to:
• determine a state of the SCM version of the repository comprising a first set of files maintained by the SCM system;
• determine an indexed state of the separate SE version of the repository using an index state query to generate an index state descriptor for a second set of files that are maintained by the SE system, the second set of files corresponding to the first set of files and including a subset of information from the first set of files;
• in response to determining that the indexed state of the SE version of the repository is not the same as the state of the SCM version of the repository:
• generate a content request, the content request comprising information defining a start state and a filter field;
• identify one or more files of the first set of files that have changed between the start state and an end state;
• filter the identified one or more files of the first set of files based on the filter field in the content request to form a filtered set of files;
• extract content and metadata for one or more files of the filtered set of files to generate a set of indexed files;
• and
• update the SE version of the repository to include the set of indexed files.

Non-transient computer readable storage comprising instructions which, when executed by a processor, cause the processor to:

• determine a state of a source code management (SCM) version of the repository comprising a first set of files maintained by the SCM system;
• determine an indexed state of a separate search engine (SE) version of the repository using an index state query to generate an index state descriptor for a second set of files that are maintained by the SE system, the second set of files corresponding to the first set of files and including a subset of information from the first set of files;
• in response to determining that the indexed state of the SE version of the repository is not the same as the state of the SCM version of the repository:
• generate a content request, the content request comprising information defining a start state and an end state of the SCM version of the repository and one or more filter fields, the start state defining a state of the search SE version of the repository and the end state defining a state of the SCM version of the repository;
• identify one or more files of the first set of files that have changed between the start state and the end state;
• filter the identified one or more files of the first set of files based on the one or more filter fields in the content request to form a filtered set of files;
• extract content and metadata for one or more files of the filtered set of files to generate a set of indexed files;
• and
• update the SE version of the repository to include the set of indexed files.