Method and system for monitoring manufacturing operations using computer vision for human performed tasks

A monitoring system for monitoring an operation, the system comprising:

• a processor;
• and
• a nontransitory computer-readable medium including instructions that are executable by the processor, wherein the instructions include:
• identifying a human operator and a controlled element within a workspace based on data acquired from sensors within the workspace at which the operation it to be performed, wherein the data includes images;
• estimating a pose of the human operator using the images;
• monitoring a state of the controlled element based on the data acquired;
• detecting a known activation trigger based on the pose estimated of the human operator, the state of the controlled element, or a combination thereof, wherein the known activation trigger indicates a start of the operation;
• tracking a duration of the operation in response to the known activation trigger being detected;
• determining whether an abnormality occurred based on the estimated pose, the state of the controlled element, the duration of the operation, or a combination thereof
• detecting a known completion trigger based on the pose estimated of the human operator, the state of the controlled element, or a combination thereof, wherein the known completion trigger indicates an end of the operation;
• determining the duration of the operation in response to the known completion trigger not being detected;
• determining whether the duration of the operation is greater than a duration benchmark parameter associated with the operation;
• and
• determining the abnormality occurred in response to the duration of the operation being greater than the duration benchmark parameter.

A method of monitoring an operation, the method comprising;

• acquiring data from sensors including images of a workspace at which the operation is to be performed;
• identifying a human operator and a controlled element within the workspace using the images;
• detecting a known activation trigger indicative of a start of the operation based on the images;
• tracking a duration of the operation in response to the known activation trigger being detected;
• estimating, in real-time, a pose of the human operator during the operation based on the images;
• determining, during the operation, whether the estimated pose is abnormal based on a pose benchmark parameter associated with the operation;
• monitoring a state of the controlled element during the operation based on the data;
• determining, during the operation, whether the state of the controlled element is abnormal based on an element benchmark parameter associated with the controlled element;
• detecting a known completion trigger indicative of an end of the operation based on the images;
• determining the duration of the operation in response to the known completion trigger not being detected;
• determining whether the duration of the operation is greater than a duration benchmark parameter associated with the operation;
• determining the abnormality occurred in response to the duration of the operation being greater than the duration benchmark parameter;
• and
• performing a remedial action in response to an abnormal operation being detected, wherein the abnormal operation includes the estimated pose being abnormal, the state of the controlled element being abnormal, the duration of the operation being greater than the duration benchmark parameter, or a combination thereof.