# US 12,602,378 B2 · Document processing and response generation system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,378 B2 - Document ID: US12602378B2 - Status: Granted - Kind: B2 - Application number: 18/435,357 - Filed: 2024-02-07 - Published: 2024-07-25 - Granted: 2026-04-14 - Earliest claimed: 2020-06-25 - Assignees: Pryon Incorporated - Applicants: Pryon Incorporated - Inventors: David Nahamoo; Igor Roditis Jablokov; Vaibhava Goel; Etienne Marcheret; Ellen Eide Kislal; Steven John Rennie; Marie Wenzel Meteer; Soonthorn Ativanichayaphong; Joseph Allen Pruitt; John Michael Pruitt; Bryan Dempsey; Rangachari Anand; Meghan Hickey; Ajinkya Jitendra Zadbuke; Neil Rohit Mallinar; Lucas C. Siler; Gregory Pelton; Chul Sung; Matthew Brian Cobb; Carmi Rothberg - Web page: https://patentgrep.com/patent/US12602378B2 ## CPC - G06F 16/24522: {Translation of natural language queries to structured queries} - G06F 16/3328: {using graphical result space presentation or visualisation} - G06F 16/3329: Natural language query formulation - G06F 16/3349: Reuse of stored results of previous queries - G06F 16/335: Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536) - G06F 16/338: Presentation of query results - G06F 16/9032: Query formulation - G06F 16/93: Document management systems - G06F 16/9574: {of access to content, e.g. by caching} - G06F 40/131: Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces - G06F 40/137: Hierarchical processing, e.g. outlines - G06F 40/151: Transformation - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06F 40/247: Thesauruses; Synonyms - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/30: Semantic analysis - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 5/04: Inference or reasoning models - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Disclosed are methods, systems, devices, apparatus, media, design structures, and other implementations, including a method that includes receiving, at a local device from a remote device, query data representative of a question relating to source content of a source document of a repository of a plurality of source documents, with the source content being associated with transformed content accessible from the local device. The method further includes generating, in response to determination, at the local device, of a match between the query data and at least one portion of the transformed content, output data comprising one or more of a pointer to access, in the source document accessible from the remote device, at least one portion of the source document corresponding to the at least one portion of the transformed content, or a copy of the at least one portion of the source document. ## Claims ### Claim 1 A method for information retrieval from documents in a document storage using a computer-implemented information-retrieval system coupled to the document storage, the method comprising: determining configuration data for a first trained neural network transformer and for a second trained neural network transformer using a first training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data according to comparisons for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data; determining configuration data for a third trained neural network transformer using a second training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data so that an output of said transformer provides an output indicative of whether the document sequence includes an answer to the query and so that an output of said transformer provides an output indicative of a span in the document sequence containing an answer to the query; ingesting a plurality of documents, including receiving the plurality of documents from the document storage, segmenting the documents into a plurality of document segments, at least some documents each being segmented into multiple segments, processing, using the computer-implemented information-retrieval system, each segment of the plurality of segments using the first trained neural network transformer to produce a corresponding fixed length numerical representation of said segment, and storing the fixed length numerical representations in association with corresponding segments; after the ingesting of the plurality of documents, receiving a query, processing the query using the second trained neural network transformer to produce a fixed length numerical representation of the query, comparing the fixed length numerical representation of the query to multiple of the fixed length numerical representations of segments of the plurality of segments, and determining one or more matching segments according to a closeness of the respective numerical representations, the one or more matching segments include a first matching segment; after determining the one or more matching segments, forming a combination from the query and at least the first matching segment, and using the third trained neural network transformer to process the combination produce a detailed output; determining a response to the query from the detailed output; and providing the response. ### Claim 2 (depends on 1) The method of claim 1, wherein the query is received from a user, and the response is provided to said user. ### Claim 3 (depends on 1) The method of claim 1, wherein the detailed output comprises a representation of a span of the response in the first matching segment, and determining of the response comprises extracting said span from the first matching segment. ### Claim 4 (depends on 1) The method of claim 1, wherein the comparing of the fixed length numerical representations comprises computing a distance between said representations, and determining of the configuration data is according to distances between for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data. ### Claim 5 (depends on 1) The method of claim 1, wherein the first trained neural network transformer and the second trained neural network transformer comprise Bidirectional Encoder Representations of Transformers (BERT) models. ### Claim 6 (depends on 1) The method of claim 1, wherein forming a combination from the query and at least the first matching segment comprises forming a concatenation of said query and said segment. ### Claim 7 (depends on 6) The method of claim 6, wherein the third neural network transformer comprises a Bidirectional Encoder Representations of Transformers (BERT) model. ### Claim 8 (depends on 1) The method of claim 1, wherein the determining the configuration data for the third trained neural network transformer further comprises updating said configuration data so that an output of said transformer provides an output indicative of a span in the document sequence containing an answer to the query. ### Claim 9 (depends on 1) The method of claim 1, wherein processing the query comprises forming a token sequence from the query, tokens in said sequence comprising words or parts of words in the query, and processing the document segments comprises forming respective token sequences comprising words or parts of words in said segments. ### Claim 10 (depends on 2) The method of claim 2, wherein: the detailed output comprises a representation of a span of the response in the first matching segment, and determining of the response comprises extracting said span from the first matching segment; the comparing of the fixed length numerical representations comprises computing a distance between said representations, and determining of the configuration data is according to distances between for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data; forming a combination from the query and at least the first matching segment comprises forming a concatenation of said query and said segment. ### Claim 11 (depends on 10) The method of claim 10, wherein: the first trained neural network transformer and the second trained neural network transformer comprise Bidirectional Encoder Representations of Transformers (BERT) models; and the third neural network transformer comprises a Bidirectional Encoder Representations of Transformers (BERT) model. ### Claim 12 A method for information retrieval comprising: configuring neural networks for use in information retrieval from documents in a document storage using a computer-implemented information-retrieval system coupled to the document storage, the configuring of the neural networks comprising: determining configuration data for a first trained neural network transformer and for a second trained neural network transformer using a first training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data according to comparisons for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data; and determining configuration data for a third trained neural network transformer using a second training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data so that an output of said transformer provides an output indicative of whether the document sequence includes an answer to the query and so that an output of said transformer provides an output indicative of a span in the document sequence containing an answer to the query; providing the configuration data for the first, second, and third trained neural network transformers for use in processing segments of documents and a query; providing instructions stored on a non-transitory machine-readable medium for the processing of the segments of the documents and the query, the instruction when executed by a data processing system cause said system to use the configuration data to perform operations including using the first trained neural network transformer to generate fixed length numerical representations of segments of documents, using the second trained neural network transformer to generate a fixed length numerical representation of the query, selecting a subset of the segments of documents based on comparison of the fixed length numerical representation of the query and the fixed length numerical representations of segments, and using the third trained neural network transformer to process a combination of the query and at least one segment in the subset of segments. ### Claim 13 (depends on 12) The method of claim 12, wherein using the third trained neural network transformer comprises producing a detailed output form the third neural network, said detailed output comprising a representation of a span of a response in the at least one segment. ### Claim 14 (depends on 12) The method of claim 12, wherein the first trained neural network transformer and the second trained neural network transformer comprise Bidirectional Encoder Representations of Transformers (BERT) models. ### Claim 15 (depends on 14) The method of claim 14, wherein the third neural network transformer comprises a Bidirectional Encoder Representations of Transformers (BERT) model. ### Claim 16 A non-transitory machine-readable medium comprising instructions stored thereon, wherein executing said instructions using a data-processing system causes said system to perform steps including: determining configuration data for a first trained neural network transformer and for a second trained neural network transformer using a first training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data according to comparisons for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data; determining configuration data for a third trained neural network transformer using a second training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data so that an output of said transformer provides an output indicative of whether the document sequence includes an answer to the query and so that an output of said transformer provides an output indicative of a span in the document sequence containing an answer to the query; ingesting a plurality of documents, including receiving the plurality of documents from a data storage, segmenting the documents into a plurality of document segments, at least some documents each being segmented into multiple segments, processing, using a computer-implemented information-retrieval system, each segment of the plurality of segments using the first trained neural network transformer to produce a corresponding fixed length numerical representation of said segment, and storing the fixed length numerical representations in association with corresponding segments; after the ingesting of the plurality of documents, receiving a query, processing the query using the second trained neural network transformer to produce a fixed length numerical representation of the query, comparing the fixed length numerical representation of the query to multiple of the fixed length numerical representations of segments of the plurality of segments, and determining one or more matching segments according to a closeness of the respective numerical representations, the one or more matching segments include a first matching segment; after determining the one or more matching segments, forming a combination from the query and at least the first matching segment, and using the third trained neural network transformer to process the combination produce a detailed output; determining a response to the query from the detailed output; and providing the response.