System and method for training a neural ode network

A non-transitory computer-readable medium on which are stored data representing instructions for training a neural network, the neural network including a neural ordinary differential equation (ODE) block, the instructions, when executed by a processor system, causing the processor system to perform the following steps:

• accessing model data defining the neural network;
• accessing a training dataset including a plurality of batches of training data;
• and
• training the neural network on the training dataset and using a first ordinary differential equation (ODE) solver;
• wherein the instructions further cause the processor system to perform accessing a second ODE solver;
• and
• wherein the training includes initializing an accuracy parameter of the first ODE solver and adjusting the accuracy parameter during the training by, during a forward pass of an iteration of the training:
• obtaining a first output of the neural network by:
• providing a layer output upstream of the neural ODE block to the first ODE solver;
• applying the first ODE solver to the neural ODE block to obtain an output of the first ODE solver;
• and
• feeding the output of the first ODE solver into a layer downstream of the neural ODE block;
• obtaining a second output of the neural network by:
• providing the layer output upstream of the neural ODE block to the second ODE solver;
• applying the second ODE solver to the neural ODE block to obtain an output of the second ODE solver;
• and
• feeding the output of the second ODE solver into the layer downstream of the neural ODE block;
• evaluating a performance function using the first output of the neural network to obtain a first performance value, the performance function quantifying a performance characteristic of the neural network with respect to a batch of training data;
• evaluating the performance function using the second output of the neural network to obtain a second performance value;
• and
• adjusting the accuracy parameter based on a difference between the first performance value and the second performance value.

A system for training a neural network, the neural network including a neural ordinary differential equation (ODE) block, the system comprising:

• a data interface configured to:
• access model data defining the neural network, and
• access a training dataset including a plurality of batches of training data;
• a processor subsystem configured to:
• train the neural network on the training dataset and using a first ODE solver,
• wherein the data interface is further configured to access a second ODE solver;
• wherein the processor subsystem is configured to train the neural network by initializing an accuracy parameter of the first ODE solver and adjusting the accuracy parameter during the training by, during a forward pass of an iteration of the training:
• obtaining a first output of the neural network by:
• providing a layer output upstream of the neural ODE block to the first ODE solver,
• applying the first ODE solver to the neural ODE block to obtain an output of the first ODE solver, and
• feeding the output of the first ODE solver into a layer downstream of the neural ODE block;
• obtaining a second output of the downstream block by:
• providing the layer output upstream of the neural ODE block to the second ODE solver,
• applying the second ODE solver to the neural ODE block to obtain an output of the second ODE solver, and
• feeding the output of the second ODE solver into the layer downstream of the neural ODE block;
• evaluating a performance function using the first output of the neural network to obtain a first performance value, the performance function quantifying a performance characteristic of the neural network with respect to a batch of training data;
• evaluating the performance function using the second output of the neural network to obtain a second performance value;
• adjusting the accuracy parameter based on a difference between the first performance value and the second performance value.