Trustworthy predictions using deep neural networks based on adversarial calibration

A computer-implemented method of training a neural network, NN, comprising the steps:

• receiving a training set T of training input data X=(X1, . . . , Xn) and corresponding ground truth data Y=(Y1, . . . , Yn) for a predetermined number C of classes, where n>1 and C>=1;
• and
• training the NN, comprising the iterative training steps:
• selecting a training sub-set B of training input data XB and corresponding ground truth data YB from the training set T, where 0<|B|<|T|;
• generating current outputs of the NN for the sub-set B by forward propagating the training input data XB of the training sub-set B in the NN;
• computing a categorical cross-entropy loss LCCE for the sub-set B based on the current outputs and the corresponding ground truth data YB of the training sub-set B;
• sampling a perturbation level εB randomly with a value from 0 to 1;
• generating an adversarial set Badv of adversarial input data Xadv by applying a perturbation randomly selected from a predefined set of perturbations and weighted with the perturbation level εB to the training input data XB of the training sub-set B, where |Xadv|=|XB|;
• generating perturbed outputs of the NN for the adversarial set Badv by forward propagating the adversarial input data Xadv of the adversarial set Badv in the NN;
• computing an adversarial calibration loss Ladv as the Euclidian norm, (L2 norm) of an expected calibration error ECE, which takes a weighted average over the perturbed outputs grouped in a predefined number M of equally spaced bins each having an associated average confidence and accuracy, where M>1;
• checking whether the training converged to a predefined lower limit for a convergence rate;
• first time updating weights of the NN based on the categorical cross-entropy loss LCCE and a predetermined training rate η, where 0<η<=1, in case the training did not converge;
• second time updating the weights of the NN based on the adversarial calibration loss Ladv weighted with a predetermined second loss factor λadv, where 0<=λadv<=1, and the predetermined training rate η, in case the training did not converge;
• and
• stopping the training of the NN in case the training converged.

A computer-implemented method of training a neural network, (NN), comprising the steps:

• receiving a training set T of training input data X=(X1, . . . , Xn) and corresponding ground truth data Y=(Y1, . . . , Yn) for a predetermined number C of classes, where n>1 and C>=1;
• and
• training the NN, comprising the iterative training steps:
• selecting a training sub-set B of training input data XB and corresponding ground truth data YB from the training set T, where 0<|B|<|T|;
• generating current outputs of the NN for the sub-set B by forward propagating the training input data XB of the training sub-set B in the NN;
• computing a categorical cross-entropy loss LCCE for the sub-set B based on the current outputs and the corresponding ground truth data YB of the training sub-set B;
• computing a predictive entropy loss LS by removing non-misleading evidence from the current outputs and distributing the remaining current outputs over the predetermined number C of classes;
• computing a combined loss L by adding to the categorical cross-entropy loss LCCE the predictive entropy loss LS weighted with a predetermined first loss factor λS, where 0<=λS<=1;
• sampling a perturbation level εB randomly with a value from 0 to 1;
• generating an adversarial set Badv of adversarial input data Xadv by applying a perturbation randomly selected from a predefined set of perturbations and weighted with the perturbation level εB to the training input data XB of the training sub-set B, where |Xadv|=|XB|;
• generating perturbed outputs of the NN for the adversarial set Badv by forward propagating the adversarial input data Xadv of the adversarial set Badv in the NN;
• computing an adversarial calibration loss Ladv as the Euclidian norm, (L2 norm) of an expected calibration error ECE, which takes a weighted average over the perturbed outputs grouped in a predefined number M of equally spaced bins each having an associated average confidence and accuracy, where M>1;
• checking whether the training converged to a predefined lower limit for a convergence rate;
• first time updating weights of the NN based on the combined loss L and a predetermined training rate η, where 0<η<=1, in case the training did not converge;
• second time updating the weights of the NN based on the adversarial calibration loss Ladv weighted with a predetermined second loss factor λadv, where 0<=λadv<=1, and the predetermined training rate η, in case the training did not converge;
• and
• stopping the training of the NN in case the training converged.