US 12,602,370 B1Granted

Detecting conflicting knowledge in data sources

AssigneeMicrosoft Technology Licensing, LLC

InventorsRoy Eisenstadt, Alexander Ostrikov, Alexander Tsvetkov

Life of this patent

Estimating the term…

Loading patent…

Abstract

A knowledge source is analyzed using machine learning models to detect conflicting or contradictory data in the documents of the knowledge source. The documents in a knowledge source are partitioned into non-overlapping, contiguous text segments containing a unique topic. Questions are generated for each text segment from a generative language model. Factoid answers are generated for each question from a question answering language model. Similar pairs of questions are identified from the answers of each question. A natural language inference model determines whether the answers to the similar questions are contradictory. A remedy is generated to address the documents having the contradictory or conflicting segments.

Claims20

Dependent claims

A system comprising:

a processor;
and
a memory that stores a program that is configured to be executed by the processor, wherein the program comprises instructions to perform acts that:
obtain a plurality of documents of a knowledge source;
partition each document of the plurality of documents of the knowledge source into a plurality of segments, wherein a segment comprises a non-overlapping, contiguous block of natural language text that pertains to a particular topic;
cause a language model to generate at least one question for each segment of the plurality of segments;
cause the language model to generate a factoid answer for the at least one question of each segment of the plurality of segments;
find a pair of similar questions from the at least one question of each segment of the plurality of segments;
cause the language model to determine whether the factoid answers for the pair of similar questions are contradictory, whereby the pair of similar questions are a contradictory pair of similar questions;
and
upon the language model determining that the pair of similar questions are a contradictory pair of similar questions, eliminate a select document associated with a select one of the contradictory pair of similar questions from the knowledge source.

8 more elements

A hardware storage device having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:

access a knowledge source comprising a plurality of documents;
identify at least one pair of documents of the plurality of documents containing conflicting information by performing actions that:
partition the plurality of documents into a plurality of segments, wherein a segment comprises a non-overlapping contiguous portion of text belonging to a topic;
cause a language model to generate a plurality of questions for each segment of the plurality of segments;
cause the language model to generate an answer for each of the plurality of questions for each segment of the plurality of segments;
identify at least one pair of similar questions from the plurality of questions for each segment of the plurality of segments, wherein a pair of similar questions is based on matching embeddings of each question of the plurality of questions or based on an edit distance of changes between pairs of questions of the plurality of questions;
and
cause the language model to determine that the answers to the at least one pair of similar questions produce results that are conflicting, whereby each of the at least one pair of similar questions is a contradictory pair of similar questions;
and
alter the knowledge source to eliminate a select document associated with a question of a contradictory pair of similar questions.

7 more elements

Dependencies are inferred from the claim text.