# US 12,602,448 B2 · Progressive neural ordinary differential equations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,448 B2
- Document ID: US12602448B2
- Status: Granted
- Kind: B2
- Application number: 17/304,163
- Filed: 2021-06-15
- Published: 2021-12-16
- Granted: 2026-04-14
- Earliest claimed: 2020-06-16
- Assignees: SRI International
- Applicants: SRI International
- Inventors: Yi Yao; Ajay Divakaran; Hammad A. Ayyubi
- Web page: https://patentgrep.com/patent/US12602448B2

## CPC

- G06F 17/13: Differential equations (using digital differential analysers G06F7/64)
- A61M 16/0493: {with means for protecting the tube from damage caused by the patient's teeth, e.g. bite block}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61M: DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY (introducing media into or onto the bodies of animals A61D7/00; means for inserting tampons A61F13/26; devices for administering food or medicines orally A61J; containers for collecting, storing or administering blood or medical fluids A61J1/05); DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY (surgery A61B; chemical aspects of surgical articles A61L); DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR {(Electrotherapy, e.g. producing anaesthesia by the use of alternating or intermittent currents A61N1/36021)}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques are described for neural networks based on Progressive Neural ODEs (PODEs). In an example, a method to progressively train a neural ordinary differential equation (NODE) model comprises processing, by a machine learning system executed by a computing system, first training data, the first training data having a first complexity, to perform training of a first layer for the NODE model; and after performing the first training, processing second training data, the second training data having a second complexity that is higher than the first complexity, to perform training of a second layer for the NODE model.

## Claims

### Claim 1

A computing system to progressively train a neural ordinary differential equation (NODE) model, the computing system comprising:
a machine learning system;
a memory configured to store the NODE model; and
processing circuitry coupled to the memory, the processing circuitry and the memory configured to execute the machine learning system, wherein the machine learning system is configured to:
process first training data, the first training data comprising irregularly spaced time series data derived from second training data by applying a complexity-reducing transformation that reduces frequency content of the second training data, to perform training of a first layer for the NODE model, the first layer comprising a first ordinary differential equation (ODE)-based transfer function parameterized by a first set of parameters; and
after processing the first training data to perform the training of the first layer for the NODE model, process, while retaining the first set of parameters of the first layer fixed, the second training data comprising irregularly spaced time series data to perform training of a second layer for the NODE model, the second layer different from the first layer and comprising a second ODE-based transfer function parameterized by a second set of parameters.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein applying the complexity-reducing transformation comprises applying at least one of a low-pass filter or principal component analysis to the second training data.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the machine learning system is configured to:
process input data to perform a prediction; and
output the prediction as output data.

### Claim 4 (depends on 3)

The computing system of claim 3, wherein the input data comprises irregularly spaced time series data having at least one of a trend and a periodicity.

### Claim 5 (depends on 3)

The computing system of claim 3,
wherein the input data comprises irregularly spaced data, and
wherein the output data comprises irregularly spaced data.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the irregularly spaced time series data of the first training data and the irregularly spaced time series data of the second training data each has at least one of a trend or a periodicity.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein the memory is configured to store an encoder, the encoder configured to:
map irregularly spaced time series data to fixed length time series data comprising a fixed length embedding; and
output the fixed length time series data as input data to the NODE model.

### Claim 8 (depends on 7)

The computing system of claim 7, wherein the machine learning system is configured to:
process the first training data to perform training of a first encoder layer of the encoder; and
after performing the training of the first layer for the NODE model, add a second encoder layer to the encoder and process the second training data to perform training of the second encoder layer of the encoder.

### Claim 9 (depends on 7)

The computing system of claim 7, wherein the memory is configured to store a decoder configured to:
map predicted values output by the NODE model to output data comprising irregularly spaced time series data; and
output the output data.

### Claim 10 (depends on 7)

The computing system of claim 7, wherein the memory is configured to store a decoder configured to:
map predicted values output by the NODE model to output data comprising irregularly spaced time series data; and
output the output data, and
wherein the machine learning system is configured to:
process the first training data to perform training of a first encoder layer of the encoder and training of a first decoder layer of the decoder; and
after performing the training of the first layer for the NODE model, add a second encoder layer to the encoder, add a second decoder layer to the decoder, and process the second training data to perform training of the second encoder layer of the encoder and training of the second decoder layer of the decoder.

### Claim 11 (depends on 1)

The computing system of claim 1, wherein the NODE model comprises a set of transfer functions that are parameterized by a set of parameters, wherein the set of transfer functions include the first ODE-based transfer function and the second ODE-based transfer function.

### Claim 12 (depends on 1)

The computing system of claim 1, wherein the machine learning system is configured to:
after performing the training of the first layer for the NODE model, add the second layer to the NODE model.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein to add the second layer to the NODE model, the machine learning system is configured to:
apply alpha blending to add the second layer to the NODE model, the alpha blending controlled by a configurable parameter.

### Claim 14

A method to progressively train a neural ordinary differential equation (NODE) model, the method comprising:
processing, by a machine learning system executed by a computing system, first training data, the first training data comprising irregularly spaced time series data derived from second training data by applying a complexity-reducing transformation that reduces frequency content of the second training data, to perform training of a first layer for the NODE model, the first layer comprising a first ordinary differential equation (ODE)-based transfer function parameterized by a first set of parameters; and
after processing the first training data to perform the training of the first layer for the NODE model, processing, while retaining the first set of parameters of the first layer fixed, the second training data comprising irregularly spaced time series data to perform training of a second layer for the NODE model, the second layer different from the first layer and comprising a second ODE-based transfer function parameterized by a second set of parameters.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
processing irregularly spaced input data to perform a prediction; and
outputting the prediction as irregularly spaced output data.

### Claim 16 (depends on 14)

The method of claim 14, wherein the irregularly spaced time series data of the first training data and the irregularly spaced time series data of the second training data each has at least one of a trend or a periodicity.

### Claim 17 (depends on 14)

The method of claim 14, further comprising:
mapping, by an encoder of the machine learning system, irregularly spaced time series data to fixed length time series data comprising a fixed length embedding; and
outputting the fixed length time series data as input data to the NODE model.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
after performing the training of the first layer for the NODE model, adding the second layer to the NODE model.

### Claim 19

Non-transitory computer readable media comprising instructions for causing processing circuitry to execute a machine learning system to progressively train a neural ordinary differential equation (NODE) model, wherein the machine learning system is configured to:
process first training data, the first training data comprising irregularly spaced time series data derived from second training data by applying a complexity-reducing transformation that reduces frequency content of the second training data, to perform training of a first layer for the NODE model, the first layer comprising a first ordinary differential equation (ODE)-based transfer function parameterized by a first set of parameters; and
after processing the first training data to perform the training of the first layer for the NODE model, process, while retaining the first set of parameters of the first layer fixed, the second training data comprising irregularly spaced time series data to perform training of a second layer for the NODE model, the second layer different from the first layer and comprising a second ODE-based transfer function parameterized by a second set of parameters.