Training a student neural network to mimic a mentor neural network with inputs that maximize student-to-mentor disagreement

A system, comprising:

• one or more processors configured to train a new neural network to mimic a target neural network, wherein the target neural network is a single neural network, without access to the target neural network or its original training dataset by:
• probing the target neural network and the new neural network with input data to generate corresponding data output by one or more layers of the respective target neural network and new neural network:
• detecting input data that generate maximum or above threshold differences between corresponding data output by the target neural network and the new neural network;
• generating a divergent probe training dataset comprising input-output pairs including the input data that generate the maximum or above threshold differences and the corresponding data output by the target neural network;
• training the new neural network to minimize differences between corresponding data output by the new neural network and the target neural network using the divergent probe training dataset detected to generate the maximum or above threshold differences in the corresponding output data between the new and target neural networks;
• and
• iteratively training the new neural network using an updated divergent probe training dataset updated to replace input-output pairs associated with output differences between the new and target neural networks that have below threshold difference measures with new input-output pairs associated with output differences between the new and target neural networks that have relatively greater difference measures, wherein the trained new neural network has a fewer number of layers and a smaller file size than the target neural network.

A non-transitory computer-readable medium comprising instructions which, when implemented in one or more processors in a computing device, cause the one or more processors to:

• train a new neural network to mimic a target neural network, wherein the target neural network is a single neural network, without access to the target neural network or its original training dataset by:
• probing the target neural network and the new neural network with input data to generate corresponding data output by one or more layers of the respective target neural network and new neural network;
• detecting input data that generate maximum or above threshold differences between corresponding data output by the target neural network and the new neural network;
• generating a divergent probe training dataset comprising input-output pairs including the input data that generate the maximum or above threshold differences and the corresponding data output by the target neural network;
• training the new neural network to minimize differences between corresponding data output by the new neural network and the target neural network using the divergent probe training dataset detected to generate the maximum or above threshold differences in the corresponding output data between the new and target neural networks;
• and
• iteratively training the new neural network using an updated divergent probe training dataset updated to replace input-output pairs associated with output differences between the new and target neural networks that have below threshold difference measures with new input-output pairs associated with output differences between the new and target neural networks that have relatively greater difference measures, wherein the trained new neural network has a fewer number of layers and a smaller file size than the target neural network.