Methods and systems for automatically creating statistically accurate ergonomics data

One or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform a machine learning based ergonomics method comprising:

• determining one or more data sets including one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of a manufacturing operation and spatio-temporal data of an actor, wherein the determining the one or more data sets comprises:
• receiving a video frame sensor stream from a video sensor monitoring the manufacturing operation;
• generating a two-dimensional array of feature vectors by performing a two-dimensional convolution operation on the video frame sensor stream with a frame extractor;
• determining a dynamic region of interest (RoI) in the video frame sensor stream with an RoI detector, wherein a convolution neural network comprises convolution layers shared by the frame extractor and the RoI detector, wherein the dynamic RoI encloses an area of a video frame of the video frame sensor stream in which a specific action is occurring;
• extracting a fixed-sized feature vector from the area within the dynamic RoI;
• and
• analyzing actions within the fixed-sized feature vector of the video frame sensor stream while discarding areas of the video frame of the video frame sensor stream outside the fixed-sized feature vector to determine the one or more data sets;
• accessing one or more ergonomic factors including a work limit, a work zone and a hazard score;
• statistically analyzing by artificial intelligence in real time the one or more data sets based on the one or more ergonomic factors to determine an ergonomic data set;
• adjusting at least one of the one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of the manufacturing operation in real time based on the ergonomic data set;
• and
• forwarding feedback in real time based on the ergonomic data set, wherein:
• the feedback is determined by applying the convolution neural network to a plurality of sliding windows such that no computations are repeated.

A system comprising:

• one or more data storage units;
• a video sensor configured to produce a video frame sensor stream;
• and
• a computing circuit configured to:
• determine one or more data sets in the one or more data storage units including one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of a manufacturing operation and spatio-temporal data of a first action, wherein the one or more data sets are determined by:
• performing a two-dimensional convolution operation on the video frame sensor stream to generate a two-dimensional array of feature vectors through a frame feature extractor;
• combining neighboring ones of the feature vectors to determine a dynamic region of interest (RoI) in the video frame sensor stream through an RoI detector, wherein the RoI detector and the frame feature extractor share layers of a convolution neural network;
• and
• extracting a fixed-sized feature vector from an area of a video frame of the video frame sensor stream within the dynamic RoI and discarding the remaining feature vectors of the video frame, wherein the convolution neural network analyzes actions in the video frame within the dynamic RoI to determine the one or more data sets;
• access one or more ergonomic factors in the one or more data storage units including a work limit, a work zone and a hazard score;
• statistically analyze in real time by one or more processing units the one or more data sets based on the one or more ergonomic factors to determine an ergonomic data set;
• store the ergonomic data set indexed to corresponding portions of the one or more data sets in the one or more data structures on the one or more data storage units;
• and
• forward feedback in real time based on the ergonomic data set, wherein:
• the convolution neural network determines the feedback in a plurality of sliding windows without repeating computations.