Sharding program compilation for large-scale static analysis

A system comprising:

• one or more processors;
• a computer-readable medium coupled to the one or more processors having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
• receiving original compilation information, including one or more groups of source files and a module path that identifies a location of module containers, the one or more groups of source files referencing modules in the module containers;
• running a compiler to determine a set of modules in the module containers that are referenced by a single source file of the groups of source files and to determine other source files that are referenced by the single source file,
• the step of running the compiler including:
• building a modified set of arguments for further source files of the source files in the one or more groups;
• compiling the single source file, with the modified set of arguments and an instrumented file manager object;
• and
• tracking, by the file manager object, which files are read while compiling;
• modifying the compilation information on the basis of the single source file, by replacing the one or more groups of source files with the single source file, by replacing the module path with a new module path, the new module path identifying the location of the referenced set of modules, and by including other source files referenced by the single source file,
• the step of modifying the compilation information including:
• building a compilation description, including the new module path, for the single source file based on files tracked by the file manager object, the new module path being a classpath, and the step of building a compilation description for the single source the including:
• receiving the list of tracked files;
• obtaining a directory containing a tracked file that is a class file;
• building a classpath to include the directory;
• obtaining a basepath of a tracked file that is a source file;
• building a sourcepath to include the basepath;
• building the compilation description for the single source file by including the single source file, the classpath, and the sourcepath;
• and
• storing the compilation description and list of tracked files as structured data in a data store;
• and
• performing the static analysis on the single source file using the compilation description and the list of tracked files stored in the data store.

A non-transitory computer-readable storage medium storing a computer program that, when executed by a computer performs steps of:

• receiving original compilation information, including one or more groups of source files and a module path that identifies a location of module containers, the one or more groups of source files referencing modules in the module containers;
• running a compiler to determine a set of modules in the module containers that are referenced by a single source file of the groups of source files and to determine other source files that are referenced by the single source file,
• the step of running the compiler including:
• building a modified set of arguments for further source files of the source files in the one or more groups;
• compiling the single source file, with the modified set of arguments and an instrumented file manager object;
• and
• tracking, by the file manager object, which files are read while compiling;
• modifying the compilation information on the basis of the single source file, by replacing the one or more groups of source files with the single source file, by replacing the module path with a new module path, the new module path identifying the location of the referenced set of modules, and by including other source files referenced by the single source file,
• the step of modifying the compilation information including:
• building a compilation description, including the new module path, for the single source file based on files tracked by the file manager object, the new module path being a classpath, and the step of building a compilation description for the single source the including:
• receiving the list of tracked files;
• obtaining a directory containing a tracked file that is a class file;
• building a classpath to include the directory;
• obtaining a basepath of a tracked file that is a source file;
• building a sourcepath to include the basepath;
• building the compilation description for the single source file by including the single source file, the classpath, and the sourcepath;
• and
• storing the compilation description and list of tracked files as structured data in a data store;
• and
• performing the static analysis on the single source file using the compilation description and the list of tracked files stored in the data store.