# US 12,008,333 B2 · Computer implemented methods for the automated analysis or use of data, including use of a large language model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,333 B2 - Document ID: US12008333B2 - Status: Granted - Kind: B2 - Application number: 18/515,488 - Filed: 2023-11-21 - Published: 2024-03-21 - Granted: 2024-06-11 - Earliest claimed: 2022-02-22 - Assignees: Unlikely Artificial Intelligence Limited - Applicants: Unlikely Artificial Intelligence Limited - Inventors: William Tunstall-Pedoe; Robert Heywood; Seth Warren; Paul Benn; Duncan Reynolds; Ayush Shah; Luci Krnic; Ziyi Zhu - Web page: https://patentgrep.com/patent/US12008333B2 ## CPC - G06F 40/56: Natural language generation - G06F 40/205: Parsing - G06F 40/30: Semantic analysis - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods are provided, such as a method of interacting with a large language model (LLM), including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt; and such as a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. Related computer systems are provided. ## Claims ### Claim 1 A computer-implemented method of automatically removing hallucinations from natural language text generated by a large language model (LLM), including the steps of: (a) providing a prompt or query to the LLM; (b) automatically generating a baseline response to the prompt or query, the baseline response including factual assertions; (c) automatically generating one or more verification questions to test the accuracy of one or more of the factual assertions for factual accuracy or inaccuracy; (d) systematically answering the or each verification questions in a manner that is not dependent on the baseline response; (e) using the answers to the or each verification question to identify one or more factual inaccuracies or hallucinations present in the baseline response; (f) automatically using the or each answer to the verification question or questions to generate a final natural language output, in which one or more factual inaccuracies or hallucinations present in the baseline response, have been removed. ### Claim 2 (depends on 1) The method of claim 1 in which there are multiple verification questions forming a sequence or chain of verification. ### Claim 3 (depends on 1) The method of claim 1 in which one or more of the verification questions do not include phrasing that matches the prompt or the baseline response. ### Claim 4 (depends on 1) The method of claim 1 in which one or more of the verification questions are simpler than the original prompt. ### Claim 5 (depends on 1) The method of claim 1 in which one or more of the verification questions are answered independently. ### Claim 6 (depends on 5) The method of claim 5 in which one or more of the verification questions do not include phrasing that matches the prompt or query, or the baseline response. ### Claim 7 (depends on 1) The method of claim 1 in which one or more of the verification questions are automatically answered as either true or false by the LLM. ### Claim 8 (depends on 1) The method of claim 1 in which one or more of the verification questions are open verification questions. ### Claim 9 (depends on 1) The method of claim 1 in which one or more of the verification questions are generated by the LLM. ### Claim 10 (depends on 9) The method of claim 9 in which the verification questions are generated by the LLM in response to a prompt. ### Claim 11 (depends on 1) The method of claim 1 in which the method includes the step of using explicit reasoning steps when cross-checking whether each of the answers to the verification questions indicate an inconsistency with the baseline response. ### Claim 12 (depends on 1) The method of claim 1 including the step of extracting from the baseline response a collection of one or more factual assertions by providing the LLM with the baseline response and a prompt asking the LLM to provide a continuation generating the factual assertions. ### Claim 13 (depends on 12) The method of claim 12 where a factual assertion derived from the baseline response is automatically translated into a verification question. ### Claim 14 (depends on 1) The method of claim 1 where, if a verification question is answered as false, then the related factual assertions derived from the draft natural language text is treated as false and is not included in the final natural language output. ### Claim 15 (depends on 1) The method of claim 1 where the sources of any previous known knowledge used to validate a factual assertion as false are returned as part of the process of checking one or more of the factual assertions for factual accuracy or inaccuracy. ### Claim 16 (depends on 1) The method of claim 1 where the checking the one or more factual assertions for factual accuracy or inaccuracy includes a further step of generating a natural language explanation of why a factual assertion is determined to be true or false. ### Claim 17 (depends on 1) The method of claim 1 in which the method includes the step of supplementing the information stored in the weights of the LLM by referencing a semantic representation of knowledge. ### Claim 18 (depends on 17) The method of claim 17 in which the semantic representation of knowledge is used to provide additional context to the LLM. ### Claim 19 (depends on 1) The method of claim 1 in which the method includes the step of supplementing the information stored in the weights of the LLM by referencing dynamic real-time information. ### Claim 20 (depends on 1) The method of claim 1 which includes the step of providing a reasoning system able to reason with a symbolic representation of the world, translating the verification questions or assertions into a representation compatible with the reasoning system and utilising the reasoning system to identify assertions which are true or false. ### Claim 21 (depends on 20) The method of claim 20 in which the reasoning system uses a universal language. ### Claim 22 A computer implemented system configured to automatically remove hallucinations from natural language text generated by a LLM, the system being configured to: (a) receive a prompt or query to the LLM; (b) automatically generate a baseline response to the prompt or query, the baseline response including factual assertions; (c) automatically generate one or more verification questions to test the accuracy of one or more of the factual assertions for factual accuracy or inaccuracy; (d) systematically answer the or each verification questions in a manner that is not dependent on the baseline response; (e) use the answers to the or each verification question to identify one or more factual inaccuracies or hallucinations present in the baseline response; (f) automatically use the or each answer to the verification question or questions to generate a final natural language output, in which one or more factual inaccuracies or hallucinations present in the baseline response, have been removed. ### Claim 23 A large language model system configured to automatically remove hallucinations from natural language text generated by the LLM system, the LLM system being configured to: (a) receive a prompt or query; (b) automatically generate a baseline response to the prompt or query, the baseline response including factual assertions; (c) automatically generate one or more verification questions to test the accuracy of one or more of the factual assertions for factual accuracy or inaccuracy; (d) systematically answer the or each verification questions in a manner that is not dependent on the baseline response; (e) use the answers to the or each verification question to identify one or more factual inaccuracies or hallucinations present in the baseline response; (f) automatically use the or each answer to the verification question or questions to generate a final natural language output, in which one or more factual inaccuracies or hallucinations present in the baseline response, have been removed.