Human-in-the-loop interactive model training

A computer-implemented method of training a predictive model from electronic health record data for a multitude of patients, the data comprising a multitude of features, each feature associated with real values and a time component, wherein the data is in a tuple format of the type {X, xi, ti} where X is the name of feature, xi is a real value of the feature and ti is a time component for the real value xi, comprising the steps of implementing the following instructions in a processor of the computer:

• a) defining a multitude of predicates as at least one of (i) binary functions operating on sequences of the tuples or (ii) logical operations on the sequences of the tuples;
• b) dividing the multitude of predicates into groups based on understandability, namely a first group of relatively more human understandable predicates and a second group of relatively less human understandable predicates;
• c) iteratively training a boosting model by performing the following:
• 1) Generating a number of new random predicates from the first group of predicates as binary functions operating on at least one of (i) sequences of the tuples or (ii) logical operations on the sequences of the tuples;
• 2) Scoring all the new random predicates by weighted information gain with respect to a class label associated with a prediction of the boosting model;
• 3) Selecting, from the new random predicates, a number of the new random predicates that are the highest with respect to their weighted information gain scores and adding them to the boosting model;
• 4) Computing weights for all the predicates in the boosting model;
• 5) Removing one or more of the selected number of the new random predicates from the boosting model in response to input from an operator;
• and
• 6) Repeating the performance of steps 1, 2, 3, 4 and 5 a plurality of times and thereby generating a final iteratively trained boosting model.

A workstation for providing operator input into iteratively training a boosting model, wherein the workstation comprises an interface and a processor, and wherein the processor is configured to perform operations comprising:

• 1) Generating a number of new random predicates as binary functions operating on at least one of (i) time sequences of input features or (ii) logical operations on the time sequences of the input features;
• 2) Scoring all the new random predicates by weighted information gain with respect to a class label associated with a prediction of the boosting model;
• 3) Selecting, from the new random predicates, a number of the new random predicates that are the highest with respect to their weighted information gain scores and adding them to the boosting model;
• 4) Computing weights for all the predicates in the boosting model;
• 5) Removing one or more of the selected number of the new random predicates from the boosting model in response to input from an operator, wherein the input is received by way of the interface;
• and
• 6) Repeating the performance of steps 1, 2, 3, 4 and 5 a plurality of times and thereby generating a final iteratively trained boosting model.

A computer-implemented method of generating a predictive model from training data, the predictive model being for predicting a label based on input data which, for each of a plurality of features X, indicates a value x of the feature at each of a plurality of times, and the training data comprising a plurality of samples, each sample indicating the value of one or more of the features at each of one of more times and a corresponding label;

• the method comprising implementing the following steps as instructions with a processor:
• defining a set of predicates, each predicate being a function which generates an output when applied to time sequences of the features or logical combinations of the time sequences of the features;
• generating a boosting model, the boosting model receiving as input the respective outputs of each of the set of predicates when applied to the samples of the training data;
• and
• performing a plurality of times, the sequence of steps of:
• (i) automatically generating a plurality of additional predicates as binary functions operating on at least one of (i) time sequences of the features or (ii) logical operations on the time sequences of the features;
• (ii) adding the plurality of additional predicates to predicates already in the boosting model to form an updated set of predicates;
• (iii) displaying a plurality of the updated set of predicates;
• and
• (iv) receiving data input rejecting one or more of the updated set of predicates;
• and
• (v) removing the rejected one or more predicates from the updated set of predicates.