Training and applying structured data extraction models

A computer system for extracting structured data from unstructured or semi-structured text in electronic documents, the computer system comprising:

• at least one memory embodying computer-readable instructions;
• and
• at least one processor coupled to the at least one memory, the at least one processor configured to execute the computer-readable instructions, which, upon execution, to cause the at least one processor to:
• receive a set of training documents;
• tokenize each training document as a training token sequence;
• render on a graphical user interface a graphical view of each training document;
• receive user input denoting:
• (i) one or more user highlighted portions of each training document and (ii) a user defined question, each highlighted portion associated with the user defined question;
• for each training document, assign the training token sequence a training label sequence based on the one or more user highlighted portions, wherein the training label sequence identifies each token as relevant to the user defined question or not relevant to the user defined question, resulting in a plurality of labelled training token sequences;
• use the plurality of labelled training token sequences to train a conditional random field (CRF) model to extract relevant information relating to the user defined question;
• tokenize text of a first electronic document as a first token sequence;
• extract features for each token of the first token sequence;
• assess, using the CRF model applied to the features of the first token sequence a first plurality of candidate label sequences for the first token sequence, wherein each first label sequence assigns a label to each token of the first token sequence, which indicates if each token of the first token sequence is relevant to the user defined question;
• identify, as having a highest first sequence probability, a first identified label sequence of the first plurality of candidate label sequences, wherein the highest first sequence probability is a probability that the first identified label sequence is correct given the first token sequence;
• identify at least one token of the first token sequence labeled by a label in the first identified label sequence to be relevant to the user defined question;
• responsive to identifying the at least one token of the first token sequence, provide the at least one token of the first token sequence as an answer to the user defined question for the first electronic document;
• tokenize text of a second electronic document as a second token sequence;
• extract features for each token of the second token sequence;
• assess, using the CRF model applied to the features of the second token sequence, a second plurality of candidate label sequences for the second token sequence, wherein each second label sequence assigns a label to each token of the second token sequence, which indicates if each token of the second token sequence is relevant to the user defined question;
• identify, as having a highest second sequence probability, a second identified label sequence of the second plurality of candidate label sequences, wherein the highest second sequence probability is a probability that the second identified label sequence is correct given the second token sequence;
• determine that the second identified label sequence indicates that no token of the second token sequence is relevant to the user defined question;
• responsive to determining that the second identified label sequence indicates that no token of the second token sequence is relevant to the user defined question, apply an answer forcing process comprising:
• determining, for each token of the second token sequence, a marginal probability of each token of the second token sequence being relevant to the question, and determining using the marginal probability of at least one token of the second token sequence a forced answer to the user defined question for the second electronic document.

Non-transitory media, embodying computer-readable instructions configured so as, upon execution by one or more computer processors, to implement operations comprising:

• receiving a set of training documents;
• tokenizing each training document as a training token sequence;
• rendering on a graphical user interface a graphical view of each training document;
• receiving user input denoting:
• (i) one or more user highlighted portions of each training document and (ii) a user defined question, each highlighted portion associated with the user defined question;
• for each training document, assigning the training token sequence a training label sequence based on the one or more user highlighted portions, wherein the training label sequence identifies each token as relevant to the user defined question or not relevant to the user defined question, resulting in a plurality of labelled training token sequences;
• using the plurality of labelled training token sequences to train a conditional random field (CRF) model to extract relevant information relating to the user defined question;
• tokenizing text of a first electronic document as a first token sequence;
• extracting features for each token of the first token sequence;
• assessing, using a CRF model applied to the features of the first token sequence a first plurality of candidate label sequences for the first token sequence, wherein each first label sequence assigns a label to each token of the first token sequence, which indicates if token of the first token sequence is relevant to the user defined question;
• identifying, as having a highest first sequence probability, a first identified label sequence of the first plurality of candidate label sequences, wherein the highest first sequence probability is a probability that the first identified label sequence is correct given the first token sequence;
• identifying at least one token of the first token sequence labeled by a label in the first identified label sequence to be relevant to the user defined question;
• responsive to identifying the at least one token of the first token sequence, providing the at least one token of the first token sequence as an answer to the user defined question for the first electronic document;
• tokenizing text of a second electronic document as a second token sequence;
• extracting features for each token of the second token sequence;
• assessing, using the CRF model applied to the features of the second token sequence, a second plurality of candidate label sequences for the second token sequence, wherein each second label sequence assigns a label to each token of the second token sequence, which indicates if each token of the second token sequence is relevant to the user defined question;
• identifying, as having a highest second sequence probability, a second identified label sequence of the second plurality of candidate label sequences, wherein the highest second sequence probability is a probability that the second identified label sequence is correct given the second token sequence;
• determining that the second identified label sequence indicates that no token of the second token sequence is relevant to the user defined question
• responsive to determining that the second identified label sequence indicates that no token of the second token sequence is relevant to the user defined question, applying an answer forcing process comprising:
• Determining, for each token of the second token sequence, a marginal probability of each token of the second sequence being relevant to the user defined question, and determining using the marginal probability of at least one token of the second token sequence a forced answer to the user defined question for the second electronic document.