Multimodal table encoding for information retrieval systems

A system comprising:

• (a) at least one hardware processor;
• and
• (b) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to, automatically:
• receive an electronic document that contains a table, wherein:
• the table comprises:
• multiple rows, multiple columns, and a schema comprising column labels or row labels, and
• the electronic document comprises a description of the table which is located externally to the table;
• operate separate machine learning encoders to separately encode the description of the table, the schema of the table, each of the rows of the table, and each of the columns of the table, respectively, the machine learning encoders including a neural network to encode the description of the table, wherein:
• the schema of the table is encoded together with end-of-column tokens or end-of-row tokens, that mark an end of each of the column or row labels, respectively,
• each of the rows of the table is encoded together with end-of-column tokens that mark an end of each data cell of the respective row, and with an end-of-row token that marks an end of the respective row, and
• each of the columns of the table is encoded together with end-of-row tokens that mark an end of each data cell of the respective column, and with an end-of-column token that marks an end of the respective column;
• apply a machine learning gating mechanism to the encoded description, encoded schema, encoded rows, and encoded columns, to produce a fused encoding of the table, wherein the fused encoding is representative of both a structure of the table and a content of the table;
• and
• store the fused encoding of the table in an index of a computerized information retrieval system.

A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to:

• receive an electronic document that contains a table, wherein:
• the table comprises:
• multiple rows, multiple columns, and a schema comprising column labels or row labels, and
• the electronic document comprises a description of the table which is located externally to the table;
• operate separate machine learning encoders to separately encode the description of the table, the schema of the table, each of the rows of the table, and each of the columns of the table, respectively, the machine learning encoders including a neural network to encode the description of the table, wherein:
• the schema of the table is encoded together with end-of-column tokens or end-of-row tokens, that mark an end of each of the column or row labels, respectively,
• each of the rows of the table is encoded together with end-of-column tokens that mark an end of each data cell of the respective row, and with an end-of-row token that marks an end of the respective row, and
• each of the columns of the table is encoded together with end-of-row tokens that mark an end of each data cell of the respective column, and with an end-of-column token that marks an end of the respective column;
• apply a machine learning gating mechanism to the encoded description, encoded schema, encoded rows, and encoded columns, to produce a fused encoding of the table, wherein the fused encoding is representative of both a structure of the table and a content of the table;
• and
• store the fused encoding of the table in an index of a computerized information retrieval system.