System and method for distributed processing of large-scale streaming data

A system for real-time distributed processing of fabrication data, comprising:

• memory configured to store a plurality of image data generated in a fabrication process in real time;
• a database configured to store a plurality of pieces of information associated with the plurality of image data in real time;
• at least one first processor implementing a master service configured to:
• monitor the database at a preset time,
• detect newly generated information after a previous monitoring time,
• select image data corresponding to the newly generated information as a processing target,
• store metadata associated with the processing target in a queue, and
• allocate metadata from the metadata associated with the processing target in the queue to a pod in response to a request from the pod;
• and
• at least one second processor implementing a plurality of pods, each pod configured to:
• operate in a keep-alive state in which a connection with the master service is kept alive,
• wherein each pod is configured to request a job determining whether there is an abnormal measurement in raw data for execution from the master service and receive allocated metadata from the master service, based on completing a previous job and entering an idle state, and
• wherein each pod is configured to access the image data stored in the memory based on the allocated metadata, and
• wherein each pod is configured to execute the job of determining whether there is an abnormal measurement associated with the raw data using a machine learning model.

A method for real-time distributed processing of fabrication data, the method being executed by one or more processor, collectively or individually, the method comprising:

• storing a plurality of raw data generated in a fabrication process, in a memory in real time;
• storing a plurality of pieces of information associated with the plurality of raw data in a database in real time;
• selecting a plurality of processing targets to be processed among the plurality of raw data, based on the plurality of pieces of information;
• storing metadata associated with each of the plurality of processing targets in a queue;
• receiving a request for executing a job of determining whether there is an abnormal measurement in the raw data from a pod, based on the pod having completed a previous job and an idle state;
• and
• providing the metadata stored in the queue to the pod to execute the job of determining whether there is an abnormal measurement in the raw data based on accessing the plurality of processing targets in the memory.