Automatic assessment of unsupervised models via trust scoring in unsupervised edge domains
AssigneeDell Products L.P.
InventorsPaulo Abelha Ferreira, Vinicius Michel Gottin, Pablo Nascimento da Silva
Life of this patent
Estimating the term…
Abstract
Model assessment is disclosed. When a model operates, tuples are transmitted to a central node. The central node can process the tuples received from multiple nodes to generate an efficiency score for the model. The efficiency score reflects how the inference of the model correlates to operator actions. Models whose assessment is below a threshold score may be retrained at least for certain classes.
Claims18
A method for assessing a quality of a performance of a model in an unsupervised manner without labelled data, the method comprising:
- receiving tuples from a node comprising one or more sensors,
- wherein each tuple includes an operator identifier of a human operator, an inference, a class, and an operator action done by the human operator,
- wherein the operator action is captured by a portion of the one or more sensors and generated by the model based on sensor data collected by the portion of the one or more sensors,
- wherein the inference is predicted by the model operating on the node based on the sensor data collected by the portion of the one or more sensors,
- wherein the inference represents an indication of an event, and
- wherein the class, for each of the tuples, is one of multiple classes and is a trajectory class which is determined from the sensor data;
- storing the tuples in a first table;
- generating a second table that associates the human operator with classes from the first table;
- generating a prediction-action table that associates the inference to the operator action using a specific class from the second table, wherein the prediction-action table captures relationships between inferences and operator actions;
- generating an efficiency score from the prediction-action table by comparing the inference and the operator action in each tuple;
- determining an efficiency score distribution for the class;
- assessing quality of the performance of the model based on the efficiency score and the efficiency score distribution;
- and
- retraining the model when the efficiency score distribution indicates that the human operator is not reacting in accordance with the inferences.
A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations for assessing a quality of a performance of a model in an unsupervised manner without labelled data, the operations comprising:
- receiving tuples from a node comprising one or more sensors,
- wherein each tuple includes an operator identifier of a human operator, an inference, a class, and an operator action done by the human operator,
- wherein the operator action is captured by a portion of the one or more sensors and generated by the model based on sensor data collected by the portion of the one or more sensors,
- wherein the inference is predicted by the model operating on the node based on the sensor data collected by the portion of the one or more sensors,
- wherein the inference represents an indication of an event, and
- wherein the class, for each of the tuples, is one of multiple classes and is a trajectory class, which is determined from the sensor data;
- storing the tuples in a first table;
- generating a second table that associates the human operator with classes from the first table;
- generating a prediction-action table that associates the inference to the operator action using a specific class from the second table, wherein the prediction-action table captures relationships between inferences and operator actions;
- generating an efficiency score from the prediction-action table by comparing the inference and the operator action in each tuple;
- determining an efficiency score distribution for the class;
- assessing the quality of the performance of the model based on the efficiency score and the efficiency score distribution;
- and
- retraining the model when the efficiency score distribution indicates that the human operator is not reacting in accordance with the inferences.
Dependencies are inferred from the claim text.