# US 12,008,473 B2 · Augmenting machine learning language models using search engine results > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,473 B2 - Document ID: US12008473B2 - Status: Granted - Kind: B2 - Application number: 18/104,210 - Filed: 2023-01-31 - Published: 2023-08-03 - Granted: 2024-06-11 - Earliest claimed: 2022-01-31 - Assignees: DeepMind Technologies Limited - Applicants: DeepMind Technologies Limited - Inventors: Angeliki Lazaridou; Elena Gribovskaya; Nikolai Grigorev; Wojciech Jan Stokowiec - Web page: https://patentgrep.com/patent/US12008473B2 ## CPC - G06N 3/08: Learning methods - G06F 16/3329: Natural language query formulation - G06F 16/953: Querying, e.g. by the use of web search engines - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/02: Neural networks - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting machine learning language models using search engine results. One of the methods includes obtaining question data representing a question; generating, from the question data, a search engine query for a search engine; obtaining a plurality of documents identified by the search engine in response to processing the search engine query; generating, from the plurality of documents, a plurality of conditioning inputs each representing at least a portion of one or more of the obtained documents; for each of a plurality of the generated conditioning inputs, processing a network input generated from (i) the question data and (ii) the conditioning input using a neural network to generate a network output representing a candidate answer to the question; and generating, from the network outputs representing respective candidate answers, answer data representing a final answer to the question. ## Claims ### Claim 1 A method performed by one or more computers, the method comprising: obtaining question data representing a question from a user; generating, from the question data, a search engine query for an Internet search engine; providing the search engine query to the Internet search engine; receiving a set of search results from the Internet search engine, wherein each search result identifies a respective document; identifying a plurality of documents from the respective documents identified by the search results; generating, from the plurality of documents, a plurality of conditioning inputs each representing at least a portion of one or more of the identified documents; generating, using a language model neural network that has been trained on a language modeling objective for a language modeling task with training data comprising text, a plurality of network outputs that each represents a respective candidate answer to the question by, for each of a plurality of the conditioning inputs, processing a respective network input comprising a respective input sequence of text tokens generated from (i) the question data and (ii) the conditioning input using the language model neural network to generate a network output representing a candidate answer to the question and comprising an output sequence of text tokens, wherein the language model neural network auto-regressively generates the output sequence by generating each particular text token in the output sequence conditioned on a current input sequence that includes the respective input sequence and any text tokens that precede the particular text token in the output sequence; generating, from the plurality of network outputs representing the respective candidate answers, answer data representing a final answer to the question; and providing the answer data to the user. ### Claim 2 (depends on 1) The method of claim 1, wherein generating, from the plurality of documents, the plurality of conditioning inputs comprises, for each of the plurality of documents: extracting, from each identified document, a text sequence; and determining a respective conditioning input from each of one or more subsequences of the extracted text sequence. ### Claim 3 (depends on 1) The method of claim 1, further comprising: for each conditioning input representing at least a portion of one or more respective documents, generating a relevance score representing a predicted relevance of at least the portion of the one or more respective documents to the question; and determining a ranking of the conditioning inputs according to the corresponding relevance scores, wherein a subset of the conditioning inputs comprises the n highest-ranked conditioning inputs in the ranking, n≥1. ### Claim 4 (depends on 3) The method of claim 3, wherein generating the relevance scores for the conditioning inputs comprises: generating an embedding of the question data; generating a respective embedding of each conditioning input; and for each conditioning input, determining a similarity between the embedding of the conditioning input and the embedding of the question data. ### Claim 5 (depends on 1) The method of claim 1, wherein for each network input generated from a respective conditioning input, the network input is further generated from one or more prompt tuples that each include (i) second question data representing a different second question, (ii) second answer data representing an answer to the respective second question, and (iii) a second conditioning input representing a document that is relevant to the respective second question. ### Claim 6 (depends on 5) The method of claim 5, wherein the language model neural network has been pre-trained on a language modeling task, and has not been trained on a question-answering task. ### Claim 7 (depends on 1) The method of claim 1, wherein generating, from the plurality of network outputs representing the respective candidate answers, answer data representing a final answer to the question comprises: generating, for each candidate answer, a respective likelihood value representing a likelihood that the candidate answer is a correct answer to the question; and determining the final answer to be the candidate answer corresponding to the highest likelihood value. ### Claim 8 (depends on 7) The method of claim 7, wherein for each candidate answer a′ to the question q corresponding to a respective conditioning input p, generating the respective likelihood value for the candidate answer a′ comprises: generating one or more initial likelihood values, the one or more initial likelihood values comprising one or more of: a first initial likelihood value representing an estimate for a probability p (a′|p, q) of the candidate answer given the conditioning input and the question, a second initial likelihood value representing an estimate for a probability p (a′|q) of the candidate answer given the question, a third initial likelihood value representing an estimate for a probability p(p|q) of the conditioning input given the question, a fourth initial likelihood value representing an estimate for a probability p (q|p) of the question given the conditioning input, or a fifth initial likelihood value representing an estimate for a probability p(q|a′, p) of the question given the candidate answer and the conditioning input; and generating the likelihood value from the one or more initial likelihood values. ### Claim 9 (depends on 8) The method of claim 8, wherein, for at least one of the initial likelihood values representing an estimate for a probability of a respective posterior element given one or more respective prior elements, the initial likelihood value is generated using the language model neural network. ### Claim 10 (depends on 9) The method of claim 9, wherein generating a particular initial likelihood value representing an estimate for a probability of a particular posterior element given one or more particular prior elements using the language model neural network comprises: determining, using a set of trained parameter values of the language model neural network, a likelihood that the language model neural network would generate a second network output representing the particular posterior element in response to processing a second network input generated from the one or more particular prior elements. ### Claim 11 (depends on 10) The method of claim 10, wherein the second network input has further been generated from one or more prompt tuples having a same particular posterior element and one or more particular prior elements as the particular initial likelihood value. ### Claim 12 (depends on 1) The method of claim 1, wherein the language model neural network comprises an autoregressive, Transformer-based neural network. ### Claim 13 (depends on 1) The method of claim 1, wherein the language model neural network is implemented on a mobile device and the method further comprises: processing the search engine query using a back-end data server; and wherein the mobile device has one or both of less working memory or less computational capacity than the back-end data server. ### Claim 14 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations comprising: obtaining question data representing a question from a user; generating, from the question data, a search engine query for an Internet search engine; providing the search engine query to the Internet search engine; receiving a set of search results from the Internet search engine, wherein each search result identifies a respective document; identifying a plurality of documents from the respective documents identified by the search results; generating, from the plurality of documents, a plurality of conditioning inputs each representing at least a portion of one or more of the identified documents; generating, using a language model neural network that has been trained on a language modeling objective for a language modeling task with training data comprising text, a plurality of network outputs that each represents a respective candidate answer to the question by, for each of a plurality of the conditioning inputs, processing a respective network input comprising a respective input sequence of text tokens generated from (i) the question data and (ii) the conditioning input using the language model neural network to generate a network output representing a candidate answer to the question and comprising an output sequence of text tokens, wherein the language model neural network auto-regressively generates the output sequence by generating each particular text token in the output sequence conditioned on a current input sequence that includes the respective input sequence and any text tokens that precede the particular text token in the output sequence; generating, from the plurality of network outputs representing the respective candidate answers, answer data representing a final answer to the question; and providing the answer data to the user. ### Claim 15 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: obtaining question data representing a question from a user; generating, from the question data, a search engine query for an Internet search engine; providing the search engine query to the Internet search engine; receiving a set of search results from the Internet search engine, wherein each search result identifies a respective document; identifying a plurality of documents from the respective documents identified by the search results; generating, from the plurality of documents, a plurality of conditioning inputs each representing at least a portion of one or more of the identified documents; generating, using a language model neural network that has been trained on a language modeling objective for a language modeling task with training data comprising text, a plurality of network outputs that each represents a respective candidate answer to the question by, for each of a plurality of the conditioning inputs, processing a respective network input comprising a respective input sequence of text tokens generated from (i) the question data and (ii) the conditioning input using the language model neural network to generate a network output representing a candidate answer to the question and comprising an output sequence of text tokens, wherein the language model neural network auto-regressively generates the output sequence by generating each particular text token in the output sequence conditioned on a current input sequence that includes the respective input sequence and any text tokens that precede the particular text token in the output sequence; generating, from the plurality of network outputs representing the respective candidate answers, answer data representing a final answer to the question; and providing the answer data to the user. ### Claim 16 (depends on 15) The system of claim 15, wherein generating, from the plurality of documents, the plurality of conditioning inputs comprises, for each of the plurality of documents: extracting, from each identified document, a text sequence; and determining a respective conditioning input from each of one or more subsequences of the extracted text sequence. ### Claim 17 (depends on 15) The system of claim 15, the operations further comprising: for each conditioning input representing at least a portion of one or more respective documents, generating a relevance score representing a predicted relevance of at least the portion of the one or more respective documents to the question; and determining a ranking of the conditioning inputs according to the corresponding relevance scores, wherein a subset of the conditioning inputs comprises the n highest-ranked conditioning inputs in the ranking, n≥1. ### Claim 18 (depends on 17) The system of claim 17, wherein generating the relevance scores for the conditioning inputs comprises: generating an embedding of the question data; generating a respective embedding of each conditioning input; and for each conditioning input, determining a similarity between the embedding of the conditioning input and the embedding of the question data. ### Claim 19 (depends on 15) The system of claim 15, wherein for each network input generated from a respective conditioning input, the network input is further generated from one or more prompt tuples that each include (i) second question data representing a different second question, (ii) second answer data representing an answer to the respective second question, and (iii) a second conditioning input representing a document that is relevant to the respective second question. ### Claim 20 (depends on 19) The system of claim 19, wherein the language model neural network has been pre-trained on a language modeling task, and has not been trained on a question-answering task.