Systems and methods for automatic job assignment

A computer implemented method of determining work task assignments within an automated production line, the method comprising:

• receiving a sensor stream at a computing device, the sensor stream comprising sensor information obtained from one or more sensors at a plurality of manufacturing stations across an assembly line;
• using the computing device executing a machine learning engine to determine one or more cycles one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters and a plurality of actors by convolution neural network deep learning from the sensor stream, the convolution neural network comprising:
• a frame feature extractor configured to perform a two-dimensional convolution operation on the sensor stream and generate a two-dimensional array of feature vectors;
• an RoI detector unit configured to combine neighboring ones of the feature vectors to determine a dynamic region of interest, wherein the RoI detector unit and the frame feature extractor share layers of the convolution neural network;
• and
• an RoI pooling unit configured to extract a fixed-sized feature vector from an area of the dynamic region of interest and discard the remaining feature vectors of the sensor stream, wherein the convolution neural network analyzes actions within the dynamic region of interest without analyzing the feature vectors outside the dynamic region of interest;
• receiving with the computing device an identity of each of the plurality of actors identified within the sensor stream;
• using the computing device executing the machine learning engine to identify a plurality of work tasks within the sensor stream that are performed by each of the plurality of actors based on the determined one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters;
• using the computing device to store, in a data structure, the sensor stream, the plurality of work tasks, and the plurality of actors;
• using the computing device to map respective work tasks performed by each of the plurality of actors to the sensor stream;
• using the computing device executing the machine learning engine to determine characterizations of the respective work tasks identified from the sensor stream performed by each of the plurality of actors including distribution functions that reflect variations in one or more of the plurality of work tasks;
• using the computing device to determine work task assignments which assign respective actors of the plurality of actors to perform respective work tasks of the plurality of work tasks during further cycles of the plurality of work tasks based on the characterizations of the plurality of work tasks for the plurality of actors;
• outputting corresponding work task assignments;
• using the computing device executing the machine learning engine to identify changes of one or more of the plurality of actors, the plurality of work tasks, and performance of the plurality of work tasks by the plurality of actors;
• using the computing device executing the machine learning engine, to update the work task assignments based on the changes of one or more of the plurality of actors, the plurality of work tasks, and performance of the plurality of work tasks by the plurality of actors;
• and
• outputting the updated work task assignments.

A system comprising:

• a processor coupled to a bus;
• one or more sensors in communication with the bus and operable to sense progress of a plurality of work tasks at one or more stations;
• and
• a memory coupled to the bus and comprising instructions of a machine learning engine that when executed by the processor cause the system to implement a method of automatically determining work task assignments, the method comprising:
• the processor executing the machine learning engine to determine one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of a manufacturing operation of a product and a plurality of actors by convolution neural network deep learning from one or more video streams of the one or more sensors, wherein the machine learning engine determines the one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters and the plurality of actors by:
• generating a two-dimensional array of feature vectors by performing a two-dimensional convolution operation on the one or more video streams with a frame extractor;
• determining a dynamic region of interest in the array of feature vectors with a RoI detector unit, wherein the convolution neural network comprises convolution layers shared by the frame extractor and the RoI detector unit;
• extracting a fixed-sized feature vector from an area within the dynamic region of interest;
• and
• analyzing actions within the fixed-sized feature vector while discarding areas of the one or more video streams outside the fixed-sized feature vector;
• the processor executing the machine learning engine to identify a plurality of work tasks performed by the plurality of actors at the one or more stations based on the one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters and the plurality of actors determined from the one or more sensors;
• the processor executing the machine learning engine to determine work task assignments of the identified plurality of work tasks for the plurality of actors;
• outputting the work task assignments;
• the processor executing the machine learning engine to identify changes of one or more of the plurality of actors, the plurality of work tasks, and performance of the plurality of work tasks by the plurality of actors, in real time;
• the processor executing the machine learning engine to update the work task assignments based on the identified changes of one or more of the plurality of actors, the plurality of work tasks, and performance of the plurality of work tasks by the plurality of actors;
• and
• outputting the updated work task assignments.