Orchestrated auxiliary pod framework for addressing scaling challenges in distributed computing systems

A computing system implementing a primary pod for processing data in a distributed computing system, the computing system comprising:

• a non-transitory memory storage comprising instructions;
• and
• a processor coupled to the non-transitory memory storage, wherein the processor executes the instructions to:
• process data received from an assigned partition in the distributed computing system, wherein the computing system comprises one or more containers that share networking and storage resources within a computing node,
• receive, from a controller that monitors metrics associated with the computing system, an instruction to transform into an orchestrator mode in response to the controller detecting a surge condition and creating one or more auxiliary pods,
• transition from processing the data to operating in the orchestrator mode,
• generate processing tasks based on the data, and
• delegate the processing tasks to the one or more auxiliary pods through a database.

A computing system implementing a controller, the computing system comprising:

• a non-transitory memory storage comprising instructions;
• and
• a processor coupled to the non-transitory memory storage, wherein the processor executes the instructions to:
• monitor metrics associated with a plurality of primary pods processing data in a distributed computing system, wherein each primary pod comprises one or more containers that share networking and storage resources within a computing node,
• detect a surge condition affecting a primary pod based on the monitored metrics,
• create one or more auxiliary pods in response to detecting the surge condition, and
• transmit an instruction to the primary pod to transform into an orchestrator mode, wherein the primary pod transitions from processing data to operating in the orchestrator mode in response to receiving the instruction, wherein the primary pod generates processing tasks based on the data, and wherein the primary pod delegates the processing tasks to the one or more auxiliary pods through a database.