Back to results
US 12,614,057 B2Granted
Training-support-based machine learning classification and regression augmentation
AssigneeThe Aerospace Corporation
InventorWalter Engevald Lillo
Life of this patent
Estimating the term…
Abstract
Machine learning models are provided that consider, during the process of producing output, various aspects of the training data and/or training process from which the models are created. A machine learning model may generate output (e.g., classification determinations or regression output) that is augmented with information regarding the distribution(s) of the corpus of training data upon which the model was trained, the features extracted from the training data, the resulting determinations made by the model, and/or other information. The augmentation may occur internally while generating the model output, or the output itself may be augmented to include distribution-based data in addition to a model output.
Claims22
A system comprising:
- computer-readable memory storing executable instructions;
- and
- one or more processors programmed by the executable instructions to at least:
- obtain a corpus of training data comprising a plurality of training data input vectors and a plurality of reference data output vectors, wherein a reference data output vector of the plurality of reference data output vectors represents a desired output generated by an artificial neural network from a corresponding training data input vector of the plurality of training data input vectors;
- train the artificial neural network using the corpus of training data to generate classification determinations;
- generate, using the artificial neural network and the plurality of training data input vectors:
- a plurality of feature vectors, wherein a feature vector of the plurality of feature vectors comprises output values of a hidden layer of the artificial neural network representing a point in a feature space, wherein the feature space is learned during training the artificial neural network to make classification determinations;
- and
- a plurality of training data output vectors, wherein a training data output vector of the plurality of training data output vectors represents a classification determination of an output layer of the artificial neural network based on a corresponding feature vector of the plurality of feature vectors;
- determine, based on an analysis of the plurality of training data output vectors with respect to the plurality of reference data output vectors:
- a first distribution of feature vectors corresponding to training data output vectors representing true positive classification determinations;
- a second distribution of feature vectors corresponding to training data output vectors representing true negative classification determinations;
- a third distribution of feature vectors corresponding to training data output vectors representing false positive classification determinations;
- a fourth distribution of feature vectors corresponding to training data output vectors representing false negative classification determinations;
- and
- a fifth distribution of feature vectors corresponding to the plurality of training data output vectors;
- wherein at least one of the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, the fourth distribution of feature vectors, or the fifth distribution of feature vectors comprises one of a Gaussian mixture model or a clustering model representing feature vectors corresponding to training data output vectors;
- and
- configure the artificial neural network to generate augmented classification output based at least partly on the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, the fourth distribution of feature vectors, and the fifth distribution of feature vectors, wherein augmented classification output comprises an indication of a degree to which an associated portion of the feature space is supported by training of the artificial neural network.
16 more elements
A system comprising:
- computer-readable memory storing executable instructions;
- and
- one or more processors programmed by the executable instructions to at least:
- obtain a corpus of training data comprising a plurality of training data input vectors and a plurality of reference data output vectors, wherein a reference data output vector of the plurality of reference data output vectors represents a desired output generated by an artificial neural network from a corresponding training data input vector of the plurality of training data input vectors;
- train the artificial neural network using the corpus of training data to generate regression output;
- generate, using the artificial neural network and the plurality of training data input vectors:
- a plurality of feature vectors, wherein a feature vector of the plurality of feature vectors comprises output values of a hidden layer of the artificial neural network representing a point in a feature space, wherein the feature space is learned during training the artificial neural network to generate regression output;
- and
- a plurality of training data output vectors, wherein a training data output vector of the plurality of training data output vectors represents a regression output of an output layer of the artificial neural network based on a corresponding feature vector;
- determine, based on an analysis of the plurality of training data output vectors with respect to the plurality of reference data output vectors:
- a first distribution of feature vectors corresponding to training data output vectors in a first subset of the training data output vectors, wherein each training data output vector in the first subset is associated with an error value that is less than a large error threshold;
- a second distribution of feature vectors corresponding to training data output vectors in a second subset of the training data output vectors, wherein each training data output vector in the second subset is associated with a positive error value having a magnitude greater than the large error threshold;
- a third distribution of feature vectors corresponding to training data output vectors in a third subset of the training data output vectors, wherein each training data output vector in the third subset is associated with a negative error value having a magnitude greater than the large error threshold;
- and
- a fourth distribution of the plurality of feature vectors;
- wherein at least one of the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, or the fourth distribution of feature vectors comprises one of a Gaussian mixture model or a clustering model representing feature vectors corresponding to training data output vectors;
- and
- configure the artificial neural network to generate augmented regression output based at least partly on the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, and the fourth distribution of feature vectors, wherein augmented regression output comprises an indication of a degree to which an associated portion of the feature space is supported by training of the artificial neural network.
15 more elements
A computer-implemented method comprising:
- under control of a computing system comprising one or more processors configured to execute specific instructions,
- obtaining a corpus of training data comprising a plurality of training data input vectors and a plurality of reference data output vectors, wherein a reference data output vector of the plurality of reference data output vectors represents a desired output generated by an artificial neural network from a corresponding training data input vector of the plurality of training data input vectors;
- training the artificial neural network using the corpus of training data to generate classification determinations;
- generating, using the artificial neural network and the plurality of training data input vectors:
- a plurality of feature vectors, wherein a feature vector of the plurality of feature vectors comprises output values of a hidden layer of the artificial neural network representing a point in a feature space, wherein the feature space is learned during training the artificial neural network to make classification determinations;
- and
- a plurality of training data output vectors, wherein a training data output vector of the plurality of training data output vectors represents a classification determination of an output layer of the artificial neural network based on a corresponding feature vector of the plurality of feature vectors;
- determining, based on an analysis of the plurality of training data output vectors with respect to the plurality of reference data output vectors:
- a first distribution of feature vectors corresponding to training data output vectors representing true positive classification determinations;
- a second distribution of feature vectors corresponding to training data output vectors representing true negative classification determinations;
- a third distribution of feature vectors corresponding to training data output vectors representing false positive classification determinations;
- a fourth distribution of feature vectors corresponding to training data output vectors representing false negative classification determinations;
- and
- a fifth distribution of feature vectors corresponding to the plurality of training data output vectors;
- wherein at least one of the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, the fourth distribution of feature vectors, or the fifth distribution of feature vectors comprises one of a Gaussian mixture model or a clustering model representing feature vectors corresponding to training data output vectors;
- and
- configuring the artificial neural network to generate augmented classification output based at least partly on the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, the fourth distribution of feature vectors, and the fifth distribution of feature vectors, wherein augmented classification output comprises an indication of a degree to which an associated portion of the feature space is supported by training of the artificial neural network.
14 more elements
A computer-implemented method comprising:
- under control of a computing system comprising one or more processors configured to execute specific instructions,
- obtaining a corpus of training data comprising a plurality of training data input vectors and a plurality of reference data output vectors, wherein a reference data output vector of the plurality of reference data output vectors represents a desired output generated by an artificial neural network from a corresponding training data input vector of the plurality of training data input vectors;
- training the artificial neural network using the corpus of training data to generate regression output;
- generating, using the artificial neural network and the plurality of training data input vectors:
- a plurality of feature vectors, wherein a feature vector of the plurality of feature vectors comprises output values of a hidden layer of the artificial neural network representing a point in a feature space, wherein the feature space is learned during training the artificial neural network to generate regression output;
- and
- a plurality of training data output vectors, wherein a training data output vector of the plurality of training data output vectors represents a regression output of an output layer of the artificial neural network based on a corresponding feature vector;
- determining, based on an analysis of the plurality of training data output vectors with respect to the plurality of reference data output vectors:
- a first distribution of feature vectors corresponding to training data output vectors in a first subset of the training data output vectors, wherein each training data output vector in the first subset is associated with an error value that is less than a large error threshold;
- a second distribution of feature vectors corresponding to training data output vectors in a second subset of the training data output vectors, wherein each training data output vector in the second subset is associated with a positive error value having a magnitude greater than the large error threshold;
- a third distribution of feature vectors corresponding to training data output vectors in a third subset of the training data output vectors, wherein each training data output vector in the third subset is associated with a negative error value having a magnitude greater than the large error threshold;
- and
- a fourth distribution of the plurality of feature vectors;
- wherein at least one of the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, or the fourth distribution of feature vectors comprises one of a Gaussian mixture model or a clustering model representing feature vectors corresponding to training data output vectors;
- and
- configuring the artificial neural network to generate augmented regression output based at least partly on the first distribution of feature vectors, the second distribution of feature vectors, the third distribution of feature vectors, and the fourth distribution of feature vectors, wherein augmented regression output comprises an indication of a degree to which an associated portion of the feature space is supported by training of the artificial neural network.
13 more elements
Dependencies are inferred from the claim text.