Multi-module and multi-task machine learning system based on an ensemble of datasets

A machine learning system, comprising:

• an input device configured to receive a plurality of training datasets comprising an image classification dataset and a natural language dataset;
• a neural network model for performing a referring expression task, including an image processing sub-module with a first plurality of neural network layers for performing an object detection task and a natural language sub-module with a second plurality of neural network layers for performing a phrase parsing task, wherein the object detection task and phrase parsing task are combined to obtain the referring expression task of the neural network model;
• a sub-module determination engine configured to:
• receive the plurality of training datasets, wherein the plurality of training datasets comprises a first training dataset comprising images and a second training dataset comprising natural language phrases;
• determine a data format associated with the first training dataset is an object detection task;
• determine a data format associated with the second training dataset is a phrase parsing task;
• identifying, using a lookup table, the image processing sub-module as being associated with the object detection task;
• identifying, using the lookup table, the natural language sub-module as being associated with the phrase parsing task;
• training the first plurality of neural network layers of the image processing sub-module using the image classification dataset, wherein the image processing sub-module is trained to perform the object detection task;
• and
• training the second plurality of neural network layers of the natural language sub-module using the natural language dataset, wherein the natural language sub-module is trained to perform the phrase parsing task;
• and
• an output device configured to output the referring expression task of the neural network model based on the image processing sub-module or the natural language sub-module.