Data storage for scalable processing of large files generated by scientific instruments

An automated method performed via a computing device for providing scientific instrument support, the method comprising:

• acquiring a first data file via one or more detectors of a scientific instrument, the first data file including unsegregated data from a plurality of scans or channels of the one or more detectors;
• applying automated processing to the first data file to generate a corresponding plurality of second data files based on one or more configuration settings, each of the second data files being smaller than the first data file and being named using a file naming convention referencing respective contents of different ones of the second data files, the corresponding plurality of the second data files being stored in an object storage;
• and
• processing a data request received from a client device, the data request being for a data portion of the first data file, the processing including providing the data portion back to the client device by accessing one or more of a first memory cache, a second memory cache, and the object storage to obtain a corresponding portion of a corresponding one of the second data files identified based on the file naming convention, the first memory cache and the second memory cache having different respective eviction policies for data loaded thereto from the object storage.

A scientific instrument, comprising:

• at least one of a mass spectrometer and a chromatography system including one or more detectors;
• and
• a computing device configured to:
• acquire a first data file via the one or more detectors, the first data file including unsegregated data from a plurality of scans or channels of the one or more detectors;
• apply automated processing to the first data file to generate a corresponding plurality of second data files based on one or more configuration settings, each of the second data files being smaller than the first data file and being named using a file naming convention referencing respective contents of different ones of the second data files, the corresponding plurality of the second data files being stored in an object storage;
• process a data request received from a client device, the data request being for a data portion of the first data file;
• and
• provide the data portion back to the client device by accessing one or more of a first memory cache, a second memory cache, and the object storage to obtain a corresponding portion of a corresponding one of the second data files identified based on the file naming convention, the first memory cache and the second memory cache having different respective eviction policies for data loaded thereto from the object storage.