Systems and methods for advance anomaly detection in a discrete manufacturing process with a task performed by a human-robot team

A method for a process control system for detecting an anomaly in a discrete manufacturing (DMP) with human-robot teams executing at least one task within the process, the method comprising steps of:

• receiving test signals that includes DMP signals from DMP sensors that includes robot operational signals, and human worker (HW) signals from human worker (HW) sensors corresponding to a human worker (HW), the HW signals including a time series of measurements of tasks completed by the HW;
• extracting a predicted sequence of events from the DMP signals and determine whether the predicted sequence of events in the DMP signals is inconsistent with a behavior of operation of the DMP described in a DMP model, and if the predicted sequence of events from the DMP signals is inconsistent with the behavior, then an alarm is to be signaled;
• extracting from the HW signals, a task completion time, a task name, measurements relating to a state of the HW and a next predicted sequenced task, and input into a Human Performance (HP) model, to obtain a state of the HW based on previously learned boundaries of the state of the HW, the state of the HW is then inputted into a Human-Robot Interaction (HRI) model, such that outputs of a Human Worker (HW) model, the HRI model, or both, are inputted into the DMP model to determine a classification of an anomaly or no anomaly,
• wherein the HP model for the HW is previously configured to have learned different states of human worker (HW) performance that correspond as a set of boundaries in the human data, and wherein the DMP model is previously configured to have learned different the operation of the manufacturing process that is used to assist in issuing classifications of anomalies or no anomaly detection;
• updating the HRI model with the robot operation signals, the HW signals and the classified anomaly;
• determining a control action of a robot interacting with the HW or a type of an anomaly alarm using the updated HRI model and the classified anomaly;
• and
• outputting the control action of the robot to change a robot action, or output the type of the anomaly alarm to a management system of the DMP, based on the updated HRI model and the classified anomaly,
• wherein the steps are implemented by a hardware processor communicatively coupled with a memory.

A non-transitory computer readable storage medium embodied thereon a program executable by a computer for performing a method for a process control system for detecting an anomaly in a discrete manufacturing process (DMP) with human-robot teams executing at least one task within the process, the method comprising steps of:

• receiving test signals that includes DMP signals from DMP sensors that includes robot operational signals, and human worker (HW) signals from human worker (HW) sensors corresponding to a human worker (HW);
• extracting a predicted sequence of events from the DMP signals and determine whether the predicted sequence of events in the DMP signals is inconsistent with a behavior of operation of the DMP described in a DMP model, and if the predicted sequence of events from the DMP signals is inconsistent with the behavior, then an alarm is to be signaled;
• extracting from the HW signals, a task completion time, measurements relating to a state of the HW and a next predicted sequenced task, and input into a Human Performance (HP) model, to obtain a state of the HW based on previously learned boundaries of the state of the HW, the state of the HW is then inputted into a Human-Robot Interaction (HRI) model, such that outputs of a Human Worker (HW) model, the HRI model, or both, are inputted into the DMP model to determine a classification of anomaly or no anomaly,
• wherein the HP model is constructed from Human worker (HW) training signals of completed training tasks during a training phase prior to receiving the HW signals, such that the HW training signals includes data for each completed training task of the completed training tasks that includes a training task name, multiple training states of the HW for the completed training task, and a next sequenced training task, such that the HW training signals and the HW signals are acquired from sensors associated with the HW during a training operation or an operation of the DMP with the human-robot teams;
• updating the HRI model with the robot operation signals, the HW signals and the classified anomaly, then determine a control action of a robot interacting with the HW or a type of an anomaly alarm using the updated HRI model and the classified anomaly;
• and
• outputting the control action of the robot to change a robot action, or output the type of the anomaly alarm to a management system of the DMP, based on the updated HRI model and the classified anomaly, wherein the steps are implemented by a hardware processor connected to a memory.

A process control system for detecting an anomaly in a discrete manufacturing process (DMP) with human-robot teams executing at least one task within the process, comprising:

• a memory configured to store data including robot data, manufacturing process (MP) data, human data, and executable models;
• an input interface configured to receive test signals including DMP signals from DMP sensors that includes robot operational signals, and human worker (HW) signals from human worker (HW) sensors corresponding to a human worker (HW);
• a hardware processor in communication with the memory and the input interface, the hardware processor configured to:
• extract a predicted sequence of events from the DMP signals, and determine whether the predicted sequence of events in the DMP signals is inconsistent with a behavior of operation of the DMP described in a DMP model, and if the predicted sequence of events from the DMP signals is inconsistent with the behavior, then an alarm is to be signaled;
• extract from the HW signals, a task completion time, measurements relating to a state of the HW and a next predicted sequenced task, and input into a Human performance (HP) model,
• wherein the HP model is constructed by a Human Task Execution (HTE) model and a model of the state of the HW, wherein the HTE model is constructed using at least one predictive model trained using Human worker (HW) training signals obtained during a training phase completing a sequence of training tasks, and wherein the model of the state of the HW is constructed using at least one classification model trained using the HW training signals, such that each completed training task is associated with multiple states of the HW, and is stored in the memory, and
• wherein the HP model determines the state of the HW based on previously learned boundaries of the state of the HW, such that the state of the HW is inputted into a Human-Robot Interaction (HRI) model, and outputs from a human worker (HW) model, the HRI model, or both, are inputted into the DMP model to determine a classification of anomaly or no anomaly;
• update the HRI model with the robot operation signals, the HW signals and the classified anomaly;
• determine a control action of a robot interacting with the HW or a type of an anomaly alarm using the updated HRI model and the classified anomaly;
• and
• an output interface to output the control action of the robot to change a robot action, or output the type of the anomaly alarm to a management system of the DMP, based on the updated HRI model and the classified anomaly.