# US 12,001,801 B2 · Question answering using dynamic question-answer database > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,801 B2 - Document ID: US12001801B2 - Status: Granted - Kind: B2 - Application number: 16/685,909 - Filed: 2019-11-15 - Published: 2021-05-20 - Granted: 2024-06-04 - Earliest claimed: 2019-11-15 - Assignees: Salesforce, Inc. - Applicants: salesforce.com, inc. - Inventors: Yuanxin Wang; Anuprit Kale; Zachary Alexander; Na Cheng - Web page: https://patentgrep.com/patent/US12001801B2 ## CPC - G06F 40/30: Semantic analysis - G06F 16/3329: Natural language query formulation - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06F 40/205: Parsing - G06N 5/041: {Abduction} - G06N 20/00: Machine learning - G06N 5/047: Pattern matching networks; Rete networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Disclosed are some implementations of systems, apparatus, methods and computer program products for integrating question generation and answer retrieval in a question answer system. The system generates a question using a set of documents and determines whether it is semantically distinct from questions in a question-answer repository. After determining that the question is semantically distinct from questions in the question-answer repository, the system adds the question to the question-answer repository. Upon receipt of a user-submitted question, the system uses the question-answer repository to identify a semantically similar question. The system retrieves an answer corresponding to the identified question from the question-answer repository and provides the answer in response to the user-submitted question. ## Claims ### Claim 1 A system comprising: a database system implemented using a server system, the database system configurable to cause: generating a question using a set of documents stored in at least one data source by at least: identifying a first one or more of the documents as being of a designated type of customer record, identifying a second one or more of the documents including a transcript of a chat between an agent and a customer associated with the customer record, and processing both the identified first one or more documents and the identified second one or more documents to identify one or more questions stored in one or more designated fields of one or more of: the identified first one or more documents, or the identified second one or more documents; comparing the generated question with a first plurality of questions within a question-answer repository such that a first plurality of similarity scores are generated, each of the first plurality of similarity scores representing a similarity between the generated question and a corresponding one of the first plurality of questions; determining whether the generated question is semantically distinct from the first plurality of questions based, at least in part, on the similarity scores; adding the generated question to the question-answer repository based, at least in part, on having determined that the generated question is semantically distinct from the first plurality of questions; and updating the set of documents to include the generated question. ### Claim 2 (depends on 1) The system as recited in claim 1, the database system further configurable to cause: parsing the set of documents to extract one or more text segments; wherein generating the question is performed using the extracted text segments. ### Claim 3 (depends on 1) The system as recited in claim 1, the database system further configurable to cause: training a question generation model using a set of question generation training data; wherein generating the question is performed using the question generation model. ### Claim 4 (depends on 1) The system as recited in claim 1, the database system further configurable to cause: obtaining a user question received from a client device; comparing the user question with a second plurality of questions within the question-answer repository such that a second plurality of similarity scores are generated, each of the second plurality of similarity scores representing a similarity between the user question and a corresponding one of the second plurality of questions; determining whether the generated question is semantically distinct from the second plurality of questions based, at least in part, on the second plurality of similarity scores; and adding the user question to the question-answer repository based, at least in part, on a result of determining whether the user question is semantically distinct from the second plurality of questions. ### Claim 5 (depends on 4) The system as recited in claim 4, the database further configurable to cause: training a similarity function using a set of similarity training data; wherein comparing the generated question with the first plurality of questions within the question-answer repository and comparing the user question with the second plurality of questions within the question-answer repository are performed using the similarity function. ### Claim 6 (depends on 1) The system as recited in claim 1, the database system further configurable to cause: obtaining an answer to the question; and updating the question-answer repository to include the answer such that the answer is associated with the question. ### Claim 7 (depends on 6) The system as recited in claim 6, wherein obtaining an answer to the question comprises: obtaining an indication of the answer from a client device; wherein updating the question-answer repository to include the answer is performed responsive to obtaining the indication of the answer from the client device. ### Claim 8 (depends on 6) The system as recited in claim 6, wherein obtaining an answer to the question comprises automatically obtaining, for the question, a corresponding answer from the set of documents or another set of documents. ### Claim 9 A method, comprising: generating a question using a set of documents stored in at least one data source by at least: identifying a first one or more of the documents as being of a designated type of customer record, identifying a second one or more of the documents including a transcript of a chat between an agent and a customer associated with the customer record, and processing both the identified first one or more documents and the identified second one or more documents to identify one or more questions stored in one or more designated fields of one or more of: the identified first one or more documents, or the identified second one or more documents; comparing the generated question with a first plurality of questions within a question-answer repository such that a first plurality of similarity scores are generated, each of the first plurality of similarity scores representing a similarity between the generated question and a corresponding one of the first plurality of questions; determining whether the generated question is semantically distinct from the first plurality of questions based, at least in part, on the similarity scores; adding the generated question to the question-answer repository based, at least in part, on having determined that the generated question is semantically distinct from the first plurality of questions; and updating the set of documents to include the generated question. ### Claim 10 (depends on 9) The method as recited in claim 9, further comprising: parsing the set of documents to extract one or more text segments; wherein generating the question is performed using the extracted text segments. ### Claim 11 (depends on 9) The method as recited in claim 9, further comprising: training a question generation model using a set of question generation training data; wherein generating the question is performed using the question generation model. ### Claim 12 (depends on 9) The method as recited in claim 9, further comprising: obtaining a user question received from a client device; comparing the user question with a second plurality of questions within the question-answer repository such that a second plurality of similarity scores are generated, each of the second plurality of similarity scores representing a similarity between the user question and a corresponding one of the second plurality of questions; determining whether the generated question is semantically distinct from the second plurality of questions based, at least in part, on the second plurality of similarity scores; and adding the user question to the question-answer repository based, at least in part, on a result of determining whether the user question is semantically distinct from the second plurality of questions. ### Claim 13 (depends on 12) The method as recited in claim 12, further comprising: training a similarity function using a set of similarity training data; wherein comparing the generated question with the first plurality of questions within the question-answer repository and comparing the user question with the second plurality of questions within the question-answer repository are performed using the similarity function. ### Claim 14 (depends on 9) The method as recited in claim 9, further comprising: obtaining an answer to the question; and updating the question-answer repository to include the answer such that the answer is associated with the question. ### Claim 15 A computer program product comprising computer-readable program code capable of being executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code comprising computer-readable instructions configurable to cause: generating a question using a set of documents stored in at least one data source by at least: identifying a first one or more of the documents as being of a designated type of customer record, identifying a second one or more of the documents including a transcript of a chat between an agent and a customer associated with the customer record, and processing both the identified first one or more documents and the identified second one or more documents to identify one or more questions stored in one or more designated fields of one or more of: the identified first one or more documents, or the identified second one or more documents; comparing the generated question with a first plurality of questions within a question-answer repository such that a first plurality of similarity scores are generated, each of the first plurality of similarity scores representing a similarity between the generated question and a corresponding one of the first plurality of questions; determining whether the generated question is semantically distinct from the first plurality of questions based, at least in part, on the similarity scores; adding the generated question to the question-answer repository based, at least in part, on having determined that the generated question is semantically distinct from the first plurality of questions; and updating the set of documents to include the generated question. ### Claim 16 (depends on 15) The computer program product as recited in claim 15, the program code further comprising computer-readable instructions configurable to cause: parsing the set of documents to extract one or more text segments; wherein generating the question is performed using the extracted text segments. ### Claim 17 (depends on 15) The computer program product as recited in claim 15, the program code further comprising computer-readable instructions configurable to cause: training a question generation model using a set of question generation training data; wherein generating the question is performed using the question generation model. ### Claim 18 (depends on 15) The computer program product as recited in claim 15, the program code further comprising computer-readable instructions configurable to cause: obtaining a user question received from a client device; comparing the user question with a second plurality of questions within the question-answer repository such that a second plurality of similarity scores are generated, each of the second plurality of similarity scores representing a similarity between the user question and a corresponding one of the second plurality of questions; determining whether the generated question is semantically distinct from the second plurality of questions based, at least in part, on the second plurality of similarity scores; and adding the user question to the question-answer repository based, at least in part, on a result of determining whether the user question is semantically distinct from the second plurality of questions. ### Claim 19 (depends on 18) The computer program product as recited in claim 18, the program code further comprising computer-readable instructions configurable to cause: training a similarity function using a set of similarity training data; wherein comparing the generated question with the first plurality of questions within the question-answer repository and comparing the user question with the second plurality of questions within the question-answer repository are performed using the similarity function. ### Claim 20 (depends on 15) The computer program product as recited in claim 15, the program code further comprising computer-readable instructions configurable to cause: obtaining an answer to the question; and updating the question-answer repository to include the answer such that the answer is associated with the question.