Method and system for retrieving information from table corpora using large language models

A computing system for extracting and displaying desired information from a set of tables, the computing system comprising:

• a storage device configured to store a set of tables including information associated with a subsurface of the Earth;
• an interface configured to receive a user query;
• and
• a processor connected to the storage device and the interface and configured to,
• select a table from the set of tables based on an embedding search performed for a question vector, which is an embedding of the user query, on a vector database of table-column questions of the set of tables,
• select one or more columns from the table based on a likelihood estimation performed in an embedding space, between (1) the user query and (2) a table summary and description of the columns for the selected table,
• determine one or more rows associated with the one or more columns, based on unique values associated with each of the one or more columns, and a similarity measure between the user query and embedded values of the unique values, in the embedding space, and
• display one or more answers in response to the user query, wherein the one or more answers includes information from the one or more columns, and the one or more rows of the selected table.

A method for generating a vector database of table-column questions for a set of tables, the method comprising:

• storing the set of tables;
• generating, with a large language model, a summary of each table of the set of tables, based on a corresponding table title, column names of each table, and additional information;
• generating, for each column of each table, a sub table containing the each column and a number of additional columns from the table, wherein rows of each column are selected to only include unique values;
• passing the sub table to the LLM for generating at least (1) a question about the sub table and the table summary and (2) a corresponding answer;
• and
• storing questions and answers generated by the LLM, for all the tables of the set of tables, to form a vector database of table-column questions.