Document processing and response generation system

A method for information retrieval comprising:

• configuring neural networks for use in information retrieval from documents in a document storage using a computer-implemented information-retrieval system coupled to the document storage, the configuring of the neural networks comprising:
• determining configuration data for a first trained neural network transformer and for a second trained neural network transformer using a first training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data according to comparisons for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data;
• and
• determining configuration data for a third trained neural network transformer using a second training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data so that an output of said transformer provides an output indicative of whether the document sequence includes an answer to the query and so that an output of said transformer provides an output indicative of a span in the document sequence containing an answer to the query;
• providing the configuration data for the first, second, and third trained neural network transformers for use in processing segments of documents and a query;
• providing instructions stored on a non-transitory machine-readable medium for the processing of the segments of the documents and the query, the instruction when executed by a data processing system cause said system to use the configuration data to perform operations including
• using the first trained neural network transformer to generate fixed length numerical representations of segments of documents,
• using the second trained neural network transformer to generate a fixed length numerical representation of the query,
• selecting a subset of the segments of documents based on comparison of the fixed length numerical representation of the query and the fixed length numerical representations of segments, and
• using the third trained neural network transformer to process a combination of the query and at least one segment in the subset of segments.

A non-transitory machine-readable medium comprising instructions stored thereon, wherein executing said instructions using a data-processing system causes said system to perform steps including:

• determining configuration data for a first trained neural network transformer and for a second trained neural network transformer using a first training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data according to comparisons for output of the first trained neural network transformer and the second trained neural network transformer for pairs of the training data;
• determining configuration data for a third trained neural network transformer using a second training data set that includes pairs of inputs, at least some of the pairs includes a query sequence and a document sequence such that the document sequence includes an answer to the query, wherein the determining includes updating said configuration data so that an output of said transformer provides an output indicative of whether the document sequence includes an answer to the query and so that an output of said transformer provides an output indicative of a span in the document sequence containing an answer to the query;
• ingesting a plurality of documents, including
• receiving the plurality of documents from a data storage,
• segmenting the documents into a plurality of document segments, at least some documents each being segmented into multiple segments,
• processing, using a computer-implemented information-retrieval system, each segment of the plurality of segments using the first trained neural network transformer to produce a corresponding fixed length numerical representation of said segment, and
• storing the fixed length numerical representations in association with corresponding segments;
• after the ingesting of the plurality of documents,
• receiving a query,
• processing the query using the second trained neural network transformer to produce a fixed length numerical representation of the query,
• comparing the fixed length numerical representation of the query to multiple of the fixed length numerical representations of segments of the plurality of segments, and
• determining one or more matching segments according to a closeness of the respective numerical representations, the one or more matching segments include a first matching segment;
• after determining the one or more matching segments,
• forming a combination from the query and at least the first matching segment, and
• using the third trained neural network transformer to process the combination produce a detailed output;
• determining a response to the query from the detailed output;
• and
• providing the response.