Performance of neural networks under distribution shift

A system of estimating an accuracy of a neural network subject to a distribution shift, the system comprising:

• an actuator configured to actuate based on commands from a computerized control system;
• one or more processors;
• and
• memory including instructions that, when executed by the one or more processors, cause the one or more processors to:
• utilize a plurality of machine learning models trained with in-distribution data, wherein the plurality of machine learning models includes a first machine learning model and a remaining number of machine learning models;
• determine in-distribution accuracies of the plurality of machine learning models by executing the plurality of machine learning models with in-distribution data;
• determine an in-distribution agreement between (i) an output of the first machine learning model executed with the in-distribution data and (ii) outputs of remaining number of machine learning models executed with the in-distribution data;
• execute the plurality of machine learning models with unlabeled out-of-distribution data;
• determine an out-of-distribution agreement between (i) an output of the first machine learning model executed with the unlabeled out-of-distribution data and (ii) outputs of the remaining number of machine learning models executed with the unlabeled out-of-distribution data;
• compare the in-distribution agreement with the out-of-distribution agreement;
• based on the comparison indicating a sufficient similarity between the in-distribution agreement and the out-of-distribution agreement, estimate an out-of-distribution accuracy of the first machine learning models with the unlabeled out-of-distribution dataset based on (i) the in-distribution accuracies, (ii) the in-distribution agreement, and (iii) the out-of-distribution agreement;
• and
• based on the estimated accuracy exceeding an accuracy threshold, command the computerized control system to actuate the actuator.

A system of estimating an accuracy of a neural network subject to a distribution shift, the system comprising:

• one or more processors programmed to:
• determine in-distribution accuracies of a plurality of machine learning models by executing on in-distribution data, wherein the plurality of machine learning models includes a first machine learning model and a remainder of machine learning models;
• determine an in-distribution agreement between (i) an output of the first machine learning model executed with the in-distribution data and (ii) outputs of the remainder of machine learning models executed with the in-distribution data;
• execute the plurality of machine learning models with unlabeled out-of-distribution data;
• determine an out-of-distribution agreement between (i) an output of the first machine learning model executed with the unlabeled out-of-distribution data and (ii) outputs of the remainder of machine learning models executed with the unlabeled out-of-distribution data;
• compare the in-distribution agreement with the out-of-distribution agreement;
• and
• estimate an out-of-distribution accuracy of the first machine learning model with the unlabeled out-of-distribution dataset based on (i) the in-distribution accuracies, (ii) the in-distribution agreement, and (iii) the out-of-distribution agreement.