# US 2022/0036181 A1 · System and method for training a neural ode network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036181 A1
- Document ID: US20220036181A1
- Status: Published application
- Kind: A1
- Application number: 17/351,772
- Filed: 2021-06-18
- Published: 2022-02-03
- Earliest claimed: 2020-07-29
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Katharina Ott; Michael Tiemann; Prateek Katiyar
- Web page: https://patentgrep.com/patent/US20220036181A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2431: Multiple classes
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented method for training a neural network including a neural ordinary differential equation (ODE) block. A first ODE solver may be used to train the neural ODE block. A second ODE solver may be used to train and verify that the neural ODE block describes an ODE as an ODE flow. During a forward pass of an iteration of training, a first performance value is obtained by applying the first ODE solver to the neural ODE block and a second performance value is obtained by applying the second ODE solver to the neural ODE block. An accuracy parameter of the first ODE solver is adjusted based on the difference between the first performance value and the second performance value.

## Claims

### Claim 1

A computer-implemented method for training a neural network, the neural network including a neural ordinary differential equation (ODE) block, the method comprising the following steps:
accessing model data defining the neural network;
accessing a training dataset including a plurality of batches of training data; and
training the neural network on the training dataset and using a first ordinary differential equation (ODE) solver;
wherein the method further comprises accessing a second ODE solver; and
wherein the training includes initializing an accuracy parameter of the first ODE solver and adjusting the accuracy parameter during the training by, during a forward pass of an iteration of the training:
obtaining a first output of the neural network by:
providing a layer output upstream of the neural ODE block to the first ODE solver;
applying the first ODE solver to the neural ODE block to obtain an output of the first ODE solver; and
feeding the output of the first ODE solver into a layer downstream of the neural ODE block;
obtaining a second output of the neural network by:
providing the layer output upstream of the neural ODE block to the second ODE solver;
applying the second ODE solver to the neural ODE block to obtain an output of the second ODE solver; and
feeding the output of the second ODE solver into the layer downstream of the neural ODE block;
evaluating a performance function using the first output of the neural network to obtain a first performance value, the performance function quantifying a performance characteristic of the neural network with respect to a batch of training data;
evaluating the performance function using the second output of the neural network to obtain a second performance value; and
adjusting the accuracy parameter based on a difference between the first performance value and the second performance value.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining the difference between the first performance value and the second performance value; and
comparing a magnitude of the difference to a threshold;
wherein the adjusting of the accuracy parameter includes adjusting the accuracy parameter based on the comparison of the magnitude of the difference to the threshold.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the accuracy parameter is a tolerance, and wherein adjusting the accuracy parameter based on the comparison of the magnitude of the difference to the threshold includes:
when the magnitude of the difference is greater than the threshold, decreasing the tolerance of the first ODE solver, and when the magnitude of the difference is below the threshold, increasing the tolerance of the first ODE solver.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein the accuracy parameter is a step size, and wherein adjusting the accuracy parameter based on the comparison of the magnitude of the difference to the threshold includes:
when the magnitude of the difference is greater than the threshold, decreasing the step size of the first ODE solver, and when the magnitude of the difference is below the threshold, increasing the step size of the first ODE solver.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the second ODE solver is of convergence order higher than the first ODE solver.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the second ODE solver is selected to have an equal or smaller expected discretization error than the first ODE solver.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the neural network is an image classifier, the image classifier being configured to accept an image as an input and to classify the image into at least one of a plurality of classes.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the accuracy parameter is adjusted periodically during the training.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the accuracy parameter is adjusted periodically during the training including a predefined number of iterations.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the accuracy parameter is adjusted aperiodically during the training.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
repeating the adjusting of the accuracy parameter until the training of the neural network is ended.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein:
the first ODE solver and/or the second ODE solver is based on at least one of a Runge-Kutta method and a multistep method; and/or
the second ODE solver is based on a method used as an embedded higher-order solver than a method of the first ODE solver.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, further comprising:
outputting model data representing the trained neural network.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1 further comprising:
receiving an image as an input to the neural network and classifying the image into at least one of a plurality of classes.

### Claim 15

A non-transitory computer-readable medium on which are stored data representing instructions for training a neural network, the neural network including a neural ordinary differential equation (ODE) block, the instructions, when executed by a processor system, causing the processor system to perform the following steps:
accessing model data defining the neural network;
accessing a training dataset including a plurality of batches of training data; and
training the neural network on the training dataset and using a first ordinary differential equation (ODE) solver;
wherein the instructions further cause the processor system to perform accessing a second ODE solver; and
wherein the training includes initializing an accuracy parameter of the first ODE solver and adjusting the accuracy parameter during the training by, during a forward pass of an iteration of the training:
obtaining a first output of the neural network by:
providing a layer output upstream of the neural ODE block to the first ODE solver;
applying the first ODE solver to the neural ODE block to obtain an output of the first ODE solver; and
feeding the output of the first ODE solver into a layer downstream of the neural ODE block;
obtaining a second output of the neural network by:
providing the layer output upstream of the neural ODE block to the second ODE solver;
applying the second ODE solver to the neural ODE block to obtain an output of the second ODE solver; and
feeding the output of the second ODE solver into the layer downstream of the neural ODE block;
evaluating a performance function using the first output of the neural network to obtain a first performance value, the performance function quantifying a performance characteristic of the neural network with respect to a batch of training data;
evaluating the performance function using the second output of the neural network to obtain a second performance value; and
adjusting the accuracy parameter based on a difference between the first performance value and the second performance value.

### Claim 16

A system for training a neural network, the neural network including a neural ordinary differential equation (ODE) block, the system comprising:
a data interface configured to:
access model data defining the neural network, and
access a training dataset including a plurality of batches of training data;
a processor subsystem configured to:
train the neural network on the training dataset and using a first ODE solver,
wherein the data interface is further configured to access a second ODE solver;
wherein the processor subsystem is configured to train the neural network by initializing an accuracy parameter of the first ODE solver and adjusting the accuracy parameter during the training by, during a forward pass of an iteration of the training:
obtaining a first output of the neural network by:
providing a layer output upstream of the neural ODE block to the first ODE solver,
applying the first ODE solver to the neural ODE block to obtain an output of the first ODE solver, and
feeding the output of the first ODE solver into a layer downstream of the neural ODE block;
obtaining a second output of the downstream block by:
providing the layer output upstream of the neural ODE block to the second ODE solver,
applying the second ODE solver to the neural ODE block to obtain an output of the second ODE solver, and
feeding the output of the second ODE solver into the layer downstream of the neural ODE block;
evaluating a performance function using the first output of the neural network to obtain a first performance value, the performance function quantifying a performance characteristic of the neural network with respect to a batch of training data;
evaluating the performance function using the second output of the neural network to obtain a second performance value;
adjusting the accuracy parameter based on a difference between the first performance value and the second performance value.
