Write optimized, distributed, scalable indexing store

A system, comprising:

• a hardware processor configured to:
• receive a batch of a plurality of different indexing updates of a data repository associated with a secondary storage system configured to perform a backup of a primary storage system;
• send at least a first portion of the batch of the plurality of different indexing updates for storage, wherein the batch of the plurality of different indexing updates is associated with a session identifier, wherein the at least the first portion of the batch of the plurality of different indexing updates are stored in an intermediate store for temporary storage wherein a metadata store manager commits the first portion of the batch of the plurality of different indexing updates from the intermediate store to a first data bucket of a metadata store in response to an occurrence of at least one commitment factor, wherein the at least one commitment factor is associated with a particular fraction of the intermediate store being full;
• receive a second portion of the batch of the plurality of different indexing updates;
• send the second portion of the batch of the plurality of different indexing updates for storage and a request to commit the second portion of the plurality of different indexing updates, wherein the metadata store manager determines that that the session identifier associated with the batch of the plurality of different indexing updates is invalid;
• receive from the metadata store manager an indication that the session identifier associated with the batch of the plurality of different indexing updates is invalid;
• establish a new communication session with the metadata store manager, wherein the new communication session is associated with a new session identifier;
• and
• send to the metadata store manager a request to commit the first portion and the second portion of the batch of the plurality of different indexing updates, wherein in response, the metadata store manager commits in a second data bucket of the metadata store the first portion and the second portion of the batch of the plurality of different indexing updates;
• and
• a memory coupled to the hardware processor and configured to provide the hardware processor with instructions.

A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

• receiving a batch of a plurality of different indexing updates of a data repository associated with a secondary storage system configured to perform a backup of a primary storage system;
• sending at least a first portion of the batch of the plurality of different indexing updates for storage, wherein the batch of the plurality of different indexing updates is associated with a session identifier, wherein the at least the first portion of the batch of the plurality of different indexing updates are stored in an intermediate store for temporary storage, wherein a metadata store manager commits the first portion of the batch of the plurality of different indexing updates from the intermediate store to a first data bucket of a metadata store in response to an occurrence of at least one commitment factor, wherein the at least one commitment factor is associated with a particular fraction of the intermediate store being full;
• receiving a second portion of the batch of the plurality of different indexing updates;
• sending the second portion of the batch of the plurality of different indexing updates for storage and a request to a commit the second portion of the plurality of different indexing updates, wherein the metadata store manager determines that the session identifier associated with the batch of the plurality of different indexing updates is invalid;
• receiving from the metadata store manager an indication that the session identifier associated with the batch of the plurality of different indexing updates is invalid;
• establishing a new communication session with the metadata store manager, wherein the new communication session is associated with a new session identifier;
• and
• sending to the metadata store manager a request to commit the first portion and the second portion of the batch of the plurality of different indexing updates, wherein in response, the metadata store manager commits in a second data bucket of the metadata store the first portion and the second portion of the batch of the plurality of different indexing updates.