Back to results
US 12,711,323 B2Granted
Question and answering on domain-specific tabular datasets
AssigneeIntuit Inc.
InventorsVignesh Thirukazhukundram Subrahma, Arnab Chakraborty, Arkadeep Banerjee, Shrutendra Harsola
Life of this patent
Estimating the term…
Abstract
Certain aspects of the disclosure provide a question and answering system for answering questions on tabular datasets. In certain aspects, answering questions with the question and answering system includes generating a sequence of operations for answering the user question using a large language model wherein each operation of the sequence of operations encodes a data operation. Answering questions further includes generating an output based on performing the sequence of operations on a tabular dataset; determining a confidence score associated with the output; and presenting the output to the user.
Claims18
A computer-implemented method for answering questions on domain-specific tabular datasets, comprising:
- receiving a user question regarding a domain-specific tabular dataset;
- generating a sequence of operations for answering the user question using a large language model, wherein each operation of the sequence of operations encodes a data operation, comprising:
- selecting, from a set of parameterized basis functions, a final data operation, wherein the final data operation generates a response to the user question, and wherein each parameterized basis function of the set of parameterized basis functions comprises an operation and at least one parameter associated with data of the domain-specific tabular dataset;
- and
- selecting, from the set of parameterized basis functions, one or more prior data operations, wherein the one or more prior data operations generate one or more prior outputs to be used by the final data operation to generate the response to the user question;
- generating an output based on performing the sequence of operations on a tabular dataset, comprising:
- for each data operation in the sequence of operations:
- determining one or more parameters associated with the data operation;
- referencing a metadata dictionary associated with the tabular dataset for each parameter of the one or more parameters associated with the data operation;
- identifying a value from the tabular dataset for each parameter of the one or more parameters associated with the data operation based on the metadata dictionary;
- and
- operating the data operation based on the identified value from the tabular dataset for each parameter of the one or more parameters;
- and
- processing each data operation in the sequence of operations;
- and
- determining a confidence score associated with the output;
- and
- generating a natural language answer to the user question based on the output and the confidence score.
15 more elements
A processing system, comprising:
- a memory comprising computer-executable instructions;
- and a processor configured to execute the computer-executable instructions and cause the processing system to:
- receive a user question regarding a domain-specific tabular dataset;
- generate a sequence of operations for answering the user question using a large language model, wherein each operation of the sequence of operations encodes a data operation, comprising:
- selecting, from a set of parameterized basis functions, a final data operation, wherein the final data operation generates a response to the user question, and wherein each parameterized basis function of the set of parameterized basis functions comprises an operation and at least one parameter associated with data of the domain-specific tabular dataset;
- and
- selecting, from the set of parameterized basis functions, one or more prior data operations, wherein the one or more prior data operations generate one or more prior outputs to be used by the final data operation to generate the response to the user question;
- generate an output based on performing the sequence of operations on a tabular dataset, comprising:
- for each data operation in the sequence of operations:
- determining one or more parameters associated with the data operation;
- referencing a metadata dictionary associated with the tabular dataset for each parameter of the one or more parameters associated with the data operation;
- identifying a value from the tabular dataset for each parameter of the one or more parameters associated with the data operation based on the metadata dictionary;
- and
- operating the data operation based on the identified value from the tabular dataset for each parameter of the one or more parameters;
- and
- processing each data operation in the sequence of operations;
- and
- determine a confidence score associated with the output;
- and
- generate a natural language answer to the user question based on the output and the confidence score.
17 more elements
A computer-implemented method for answering questions on domain-specific tabular datasets, comprising:
- receiving a user question regarding a domain-specific tabular dataset;
- generating a sequence of operations for answering the user question using a large language model, wherein each operation of the sequence of operations encodes a data operation, comprising:
- selecting, from a set of parameterized basis functions, a final data operation, wherein the final data operation generates a response to the user question, and wherein each parameterized basis function of the set of parameterized basis functions comprises an operation and at least one parameter associated with data of the domain-specific tabular dataset;
- and
- selecting, from the set of parameterized basis functions, one or more prior data operations, wherein the one or more prior data operations generate one or more prior outputs to be used by the final data operation to generate the response to the user question;
- generating an output based on performing the sequence of operations on a tabular dataset, comprising:
- for each data operation in the sequence of operations:
- determining one or more parameters associated with the data operation;
- referencing a metadata dictionary associated with the tabular dataset for each parameter of the one or more parameters associated with the data operation;
- identifying a value from the tabular dataset for each parameter of the one or more parameters associated with the data operation based on the metadata dictionary;
- and
- operating the data operation based on the identified value from the tabular dataset for each parameter of the one or more parameters;
- and
- processing each data operation in the sequence of operations;
- determining a confidence score associated with the output, comprising:
- converting the sequence of operations to a reference question;
- generating a first embedding vector representing the reference question;
- generating a second embedding vector representing the user question;
- and
- generating the confidence score based on a difference between the first embedding vector and the second embedding vector;
- and
- generating a natural language answer to the user question based on the output and the confidence score, wherein the confidence score is provided with the natural language answer.
19 more elements
Dependencies are inferred from the claim text.