# US 2025/0094790 A1 · Neural network temporal domain generalization method and system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094790 A1
- Document ID: US20250094790A1
- Status: Published application
- Kind: A1
- Application number: 18/891,940
- Filed: 2024-09-20
- Published: 2025-03-20
- Earliest claimed: 2023-09-20
- Applicants: Royal Bank of Canada
- Inventors: Sepidehsadat HOSSEINI; Mengyao ZHAI; Hossein HAJIMIRSADEGHI; Frederick TUNG
- Web page: https://patentgrep.com/patent/US20250094790A1

## CPC

- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and techniques for neural network temporal domain generalization involve training a backbone neural network using a combination of source domains, determining a domain-specific prompt for each of the source domains while the backbone network is frozen, and sequentially determining i) temporal prompts and ii) a general prompt, while training a temporal prompt generator neural network and keeping the backbone network frozen. The various source domains are indexed temporally and respectively are made of data having a time-dependent distribution shift. The temporal prompts capture the dynamics associated with temporal drift in the data, while the general prompt captures general information across all the source domains. This allows the backbone neural network to be adapted to different time periods.

## Claims

### Claim 1

A neural network temporal domain generalization method, the method comprising:
(a) for each of multiple source domains respectively corresponding to different times and having a time-dependent distribution shift, determining a domain-specific prompt for the source domain using a backbone neural network and at least one input and at least one output from the source domain, wherein the backbone neural network is trained using a combination of the source domains and is frozen after training, and wherein the domain-specific prompt and the at least one input are input to the backbone neural network and the at least one output is output by the backbone neural network;
(b) for each of the source domains, determining a domain-specific prompt for the source domain using the backbone neural network and at least one input and at least one output from the source domain, wherein the domain-specific prompt and the at least one input are input to the backbone neural network and the at least one output is output by the backbone neural network; and
(c) sequentially determining for each of the source domains except a first one of the source domains:
(i) a temporal prompt for the source domain; and
(ii) a general prompt common to all of the source domains,
wherein the temporal prompt for the source domain and the general prompt are determined using the backbone neural network, a temporal prompt generator neural network used in respect of all of the source domains, the at least one input and the at least one output from the source domain, and at least the domain-specific prompt of a prior indexed one of the source domains,
wherein the temporal prompt is an output of the temporal prompt generator neural network used in respect of all of the source domains,
wherein the temporal prompt generator neural network is trained during generation of the temporal prompt, and
wherein each of the backbone neural network and the temporal prompt generator neural network comprises a transformer.

### Claim 2 (depends on 1)

The method of claim 1, further comprising training the backbone neural network using the combination of source domains.

### Claim 3 (depends on 2)

The method of claim 2, wherein the backbone neural network is trained to maximize a likelihood θ(Y1:τ|X1:τ), wherein the backbone neural network is parameterized by θ, and X1:τ and Y1:τ respectively represent inputs and outputs across the source domains.

### Claim 4 (depends on 1)

The method of claim 1, wherein the domain-specific prompt is determined by maximizing a likelihood θ(Yt|[PSt; Xt]) while the backbone neural network is frozen, wherein the backbone neural network is parameterized by θ, PSt is the domain-specific prompt, and Xt and Yt respectively represent inputs and outputs of the source domain specific to the domain-specific prompt.

### Claim 5 (depends on 1)

The method of claim 1, wherein the general prompt and the temporal prompt are determined by maximizing a likelihood θ(Yt|[PTt; PG; Xt]) while the backbone neural network is frozen, wherein the backbone neural network is parameterized by θ, PTt is the temporal prompt for a given one of the source domains, PG is the general prompt, and Xt and Yt respectively represent inputs and outputs of the given one of the source domains.

### Claim 6 (depends on 1)

The method of claim 1, wherein the transformer of the temporal prompt generator neural network comprises a single encoder layer.

### Claim 7 (depends on 1)

The method of claim 1, wherein the time-dependent distribution shift is continuous over all of the source domains.

### Claim 8 (depends on 1)

The method of claim 1, wherein the domain-specific prompt is prepended or appended to the input when input to the backbone neural network.

### Claim 9 (depends on 1)

The method of claim 1, wherein the first one of the source domains corresponds to the source domain earliest in time, and wherein the prior indexed one of the source domains is the source domain that immediately precedes the source domain for which the temporal prompt is being determined.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first one of the source domains corresponds to the source domain latest in time, and wherein the prior indexed one of the source domains is the source domain that immediately follows the source domain for which the temporal prompt is being determined.

### Claim 11 (depends on 1)

The method of claim 1, wherein determining the temporal prompt for the source domain comprises keeping frozen all of the temporal prompts for all of the prior indexed ones of the source domains.

### Claim 12 (depends on 1)

The method of claim 1, wherein the source domains correspond to non-overlapping periods of time.

### Claim 13 (depends on 1)

The method of claim 1, wherein an input to the backbone neural network for any one of the source domains during the sequential determining comprises the at least one input prepended or appended to the general prompt, and wherein the at least one input and the general prompt are prepended or appended to the temporal prompt.

### Claim 14 (depends on 1)

The method of claim 1, wherein the domain-specific prompts of all of the prior indexed ones of the source domains are used during the sequential determining of the temporal prompt for each of the source domains.

### Claim 15 (depends on 14)

The method of claim 14, wherein the training of the temporal prompt generator neural network, and the determining of the temporal prompt for each of the source domains and the general prompt, are performed by applying backpropagation based on a loss determined using an output of the backbone neural network.

### Claim 16 (depends on 1)

The method of claim 1, wherein the domain-specific prompts for the source domains are free parameters.

### Claim 17 (depends on 1)

The method of claim 1, further comprising determining a target output from a target input, wherein the target input and target output comprise part of a target domain that is subsequent to a last of the source domains, wherein determining the target output comprises:
(a) determining a target temporal prompt using the domain-specific prompts of the source domains; and
(b) inputting the target temporal prompt, the target input, and the general prompt to the backbone neural network.

### Claim 18

A neural network temporal domain generalization system, the system comprising at least one processing unit configured to perform a method comprising:
(a) for each of multiple source domains respectively corresponding to different times and having a time-dependent distribution shift, determining a domain-specific prompt for the source domain using a backbone neural network and at least one input and at least one output from the source domain, wherein the backbone neural network is trained using a combination of the source domains and is frozen after training, and wherein the domain-specific prompt and the at least one input are input to the backbone neural network and the at least one output is output by the backbone neural network;
(b) for each of the source domains, determining a domain-specific prompt for the source domain using the backbone neural network and at least one input and at least one output from the source domain, wherein the domain-specific prompt and the at least one input are input to the backbone neural network and the at least one output is output by the backbone neural network; and
(c) sequentially determining for each of the source domains except a first one of the source domains:
(i) a temporal prompt for the source domain; and
(ii) a general prompt common to all of the source domains,
wherein the temporal prompt for the source domain and the general prompt are determined using the backbone neural network, a temporal prompt generator neural network used in respect of all of the source domains, the at least one input and the at least one output from the source domain, and at least the domain-specific prompt of a prior indexed one of the source domains,
wherein the temporal prompt is an output of the temporal prompt generator neural network used in respect of all of the source domains,
wherein the temporal prompt generator neural network is trained during generation of the temporal prompt, and
wherein each of the backbone neural network and the temporal prompt generator neural network comprises a transformer.

### Claim 19 (depends on 18)

The system of claim 18, further comprising at least one database storing the source domains, and wherein the at least one processing unit is further configured to train the backbone neural network using the combination of source domains.

### Claim 20

At least one non-transitory computer readable medium having stored thereon computer code that is executable by at least one processor and that, when executed by the at least one processor, performs a method comprising:
(a) for each of multiple source domains respectively corresponding to different times and having a time-dependent distribution shift, determining a domain-specific prompt for the source domain using a backbone neural network and at least one input and at least one output from the source domain, wherein the backbone neural network is trained using a combination of the source domains and is frozen after training, and wherein the domain-specific prompt and the at least one input are input to the backbone neural network and the at least one output is output by the backbone neural network;
(b) for each of the source domains, determining a domain-specific prompt for the source domain using the backbone neural network and at least one input and at least one output from the source domain, wherein the domain-specific prompt and the at least one input are input to the backbone neural network and the at least one output is output by the backbone neural network; and
(c) sequentially determining for each of the source domains except a first one of the source domains:
(i) a temporal prompt for the source domain; and
(ii) a general prompt common to all of the source domains,
wherein the temporal prompt for the source domain and the general prompt are determined using the backbone neural network, a temporal prompt generator neural network used in respect of all of the source domains, the at least one input and the at least one output from the source domain, and at least the domain-specific prompt of a prior indexed one of the source domains,
wherein the temporal prompt is an output of the temporal prompt generator neural network used in respect of all of the source domains,
wherein the temporal prompt generator neural network is trained during generation of the temporal prompt, and
wherein each of the backbone neural network and the temporal prompt generator neural network comprises a transformer.
