# US 2021/0342684 A1 · Method and system for table retrieval using multimodal deep co-learning with helper query-dependent and query-independent relevance labels > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0342684 A1 - Document ID: US20210342684A1 - Status: Published application - Kind: A1 - Application number: 16/861,248 - Filed: 2020-04-29 - Published: 2021-11-04 - Earliest claimed: 2020-04-29 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Roee Shraga; Haggai Roitman; Guy Feigenblat; Mustafa Canim - Web page: https://patentgrep.com/patent/US20210342684A1 ## CPC - G06N 3/08: Learning methods - G06F 16/2282: {Tablespace storage structures; Management thereof} - G06F 16/243: {Natural language query formulation} - G06F 16/24578: {using ranking} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system and a computer-implemented method for ranking tabular data entities by likelihood of comprising answers for (natural language) queries, based on multimodal descriptions of the tabular data entities, comprising separate representations, which represent different aspects of the tabular data entities. The ranking is based on joint representations, generated from the query representation and separate representations of the tabular data entities' aspects, using gated multimodal units. The computer-implemented method may be used for applications such as web searches, data aggregation, and research tasks. ## Claims ### Claim 1 A system for ranking tabular data entities by likelihood of comprising answers for (natural language) queries, the system comprising: at least one memory storing a machine learning based model and a code; and a processor adapted to execute the code for: receiving a query and a plurality of descriptions of tabular entities, each description comprising a plurality of separate representations, and each of the plurality of separate representations represents a tabular data aspect; generating a joint representation by combining a plurality of combined representations from the plurality of separate representations, wherein the combining is executed by at least one gated multimodal unit; and generating a ranking for at least one of the tabular entities descriptions by combining the query and the joint representation. ### Claim 2 (depends on 1) The system of claim 1, wherein combining the query and the joint representation is executed by at least one additional gated multimodal unit. ### Claim 3 (depends on 1) The system of claim 1, wherein the plurality of combined representations comprises three or more separate representations. ### Claim 4 (depends on 1) The system of claim 1, wherein the machine learning based model comprises a neural network. ### Claim 5 (depends on 1) The system of claim 1, wherein at least one representation from the plurality of separate representations is obtained by embedding, generated by a neural embedding model. ### Claim 6 (depends on 1) The system of claim 1, wherein generating the ranking for the at least one of the tabular entities descriptions is further based on a prediction generated by the at least one of the plurality of separate representations. ### Claim 7 (depends on 1) The system of claim 1, wherein generating the ranking for the at least one of the tabular entities descriptions is further based on combining the at least one of the plurality of separate representations and a representation generated for the query. ### Claim 8 (depends on 6) The system of claim 6 wherein generating the ranking for the at least one of the tabular entities descriptions is further based on a relevance label for the separate prediction, wherein the predicting a relevance label was trained using at least one other prediction generated based on the plurality of separate representations. ### Claim 9 A computer implemented method for ranking tabular data entities by likelihood of comprising answers for (natural language) queries, comprising: receiving a query and a plurality of descriptions of tabular entities, each description comprising a plurality of separate representations, and each of the plurality of separate representations represents a tabular data aspect; generating a joint representation by combining a plurality of combined representations from the plurality of separate representations, wherein the combining is executed by at least one gated multimodal unit; and generating a ranking for at least one of the tabular entities descriptions by combining the query and the joint representation. ### Claim 10 A computer implemented method of training a tabular data entities ranking system, the ranking is by likelihood of comprising answers for (natural language) queries, and the training comprising: receiving a query and a plurality of descriptions of tabular entities, each description comprising a plurality of separate representations, and each of the plurality of separate representations represents a tabular data aspect; generating a joint representation by combining a plurality of combined representations from the plurality of separate representations, wherein the combining is executed by at least one gated multimodal unit; generating a ranking for at least one of the tabular entities descriptions by combining the query and the joint representation; combining at least two representations from the plurality of separate representations for calculating of a joint loss; and training at least one part of the tabular data entities ranking system using a loss function comprising the joint loss. ### Claim 11 (depends on 10) The method of claim 10, wherein combining the query and the joint representation is executed by at least one additional gated multimodal unit. ### Claim 12 (depends on 10) The method of claim 10, wherein the tabular data entities ranking system comprises a neural network. ### Claim 13 (depends on 10) The method of claim 10, wherein at least one representation of the plurality of separate representations is obtained by embedding, generated by a neural embedding model. ### Claim 14 (depends on 13) The method of claim 13, wherein the neural embedding model comprises a multi-dimensional convolutional neural network. ### Claim 15 (depends on 10) The method of claim 10, wherein the calculating of a joint loss further comprises processing a representation generated for the query and one of the plurality of separate representations using a further additional gated multimodal unit. ### Claim 16 (depends on 10) The method of claim 10, wherein the combining at least two representations is obtained by at least one further additional gated multimodal unit. ### Claim 17 (depends on 10) The method of claim 10, wherein the calculating of a joint loss further comprises an aspect loss generated for at least one of the plurality of separate representations, wherein the aspect loss is based on a prediction generated by the at least one of the plurality of separate representations. ### Claim 18 (depends on 10) The method of claim 10, wherein a separate prediction is generated based on at least one separate representations from the plurality of separate representations, wherein the separate prediction is trained to predict the tabular data entities ranking. ### Claim 19 (depends on 10) The method of claim 10, wherein a separate prediction is generated based on at least one separate representations from the plurality of separate representations and a representation generated for the query, wherein the separate prediction is trained to predict the tabular data entities ranking. ### Claim 20 (depends on 19) The method of claim 19, further comprising predicting a relevance label for the separate prediction, wherein the predicting a relevance label is trained using at least one other prediction generated based on the plurality of separate representations.