Image and semantic based table recognition

A non-transitory computer-readable medium storing executable instructions embodied thereon, which, when executed by a processing device, cause the processing device to perform operations comprising:

• determining, using a table recognition machine learning model, a first set of features based on an input image depicting a table;
• determining, based on the first set of features, a first set of predictions representing rows and columns associated with the table;
• determining, based on the first set of features, semantic information included in the table and a set of bounding boxes corresponding to the semantic information;
• determining, using the table recognition machine learning model, a set of labels associated with cells of the table based on the first set of feature, the first set of predictions, the semantic information, and the set bounding boxes;
• and
• causing the table recognition machine learning model to execute a self-supervised pre-training task based on the table by at least causing the table recognition machine learning model to predict at least a portion of the semantic information based on a modified image of the table generated by at least masking a set of pixels of the image corresponding to a cell of the table.

A system comprising:

• a memory component;
• and
• a processing device coupled to the memory component, the processing device to perform operations comprising:
• pre-training a table recognition machine learning model by at least causing the table recognition machine learning model to perform a set of self-supervision tasks to generate a pre-trained table recognition machine learning model, wherein the set of self-supervision tasks includes at least:
• predicting semantic values corresponding to a cell of a table based on a set of masked pixels included in an image of the table corresponding to the cell;
• predicting a set of column and a set of row associated with the table based on modifications to rows and columns of the table;
• and
• identifying a rotated angle associated with a table based on a rotated image of the table;
• training the pre-trained table recognition machine learning model by at least fine tuning the pre-trained table recognition machine learning model to generate a trained table recognition machine learning model;
• and
• using the trained table recognition machine learning model to perform a task.