# US 2021/0241156 A1 · System and method for machine learning architecture with adaptive importance sampling with normalizing flows

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241156 A1
- Document ID: US20210241156A1
- Status: Published application
- Kind: A1
- Application number: 17/163,106
- Filed: 2021-01-29
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Royal Bank of Canada
- Applicants: Royal Bank of Canada
- Inventors: Zhexin LAI; Amir H. KHOSHAMAN; Marcus A. BRUBAKER
- Web page: https://patentgrep.com/patent/US20210241156A1

## CPC

- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06F 17/10: Complex mathematical operations {(function generation by table look-up G06F1/03; evaluation of elementary functions by calculation G06F7/544)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 20/00: Machine learning
- G06N 3/047: Probabilistic or stochastic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system for computational estimation sampling from non-trivial probability distributions. The system comprises a processor, operating in conjunction with computer memory. The processor is configured to conduct importance sampling using normalizing flows where a base distribution has a set of parameters that can be adjusted to account for heavy-tailed distributions.

## Claims

### Claim 1

A computer system for generating a Monte Carlo estimation value from a target distribution p(x) using a proposal distribution q(x) representative of a learned importance distribution to encourage computational efficiency of the generation of the Monte Carlo estimation value, the computer system comprising:
one or more processors configured to:
instantiate a proposal distribution model data object having a first set of parameters representative of adjustable features of the proposal distribution q(x);
perform an iterative exploitation training stage for training the proposal distribution model data object, the iterative exploitation training stage including:
iteratively obtaining one or more new sample observations from the proposal distribution q(x) using a normalizing flow model to represent the learned importance distribution, wherein the normalizing flow model is defined using a succession of bi-jective transformations conducted on a random variable z∈D having an base distribution pz(z) representative of a probability density function of the random variable z, the base distribution having a second set of parameters, including a degree of freedom parameter adapted for modelling tail-behavior of the target distribution p(x);
for each new sample observation of the one or more new sample observations, iteratively determining a corresponding weight value based at least upon a ratio of the target distribution p(x) and the proposal distribution q(x) corresponding to the respective new sample observation;
updating both the first set of parameters representative of adjustable features of the proposal distribution q(x) and the second set of parameters of the base distribution based on the one or more new sample observations such that the proposal distribution q(x) is updated for a next iteration of the exploitation training stage;
recording the one or more new sample observations from the proposal distribution q(x) along with the corresponding weight values; and
continuing to alternatively obtain new sample observations and update both the first set of parameters of the proposal distribution q(x) and the second set of parameters of the base distribution until a number of exploitation training steps are taken; and
generate the Monte Carlo estimation value based at least on a sum of the recorded weight values applied to each of the recorded sample observations; and
store the Monte Carlo estimation value as a data value in an output data structure.

### Claim 2 (depends on 1)

The computer system of claim 1, wherein the proposal distribution q(x) is initially poorly fit to the target distribution p(x), and the one or more processors are further configured to:
perform an iterative exploration training stage prior to the iterative exploitation training stage, the iterative exploration training stage including, iteratively:
iteratively obtaining a new sample observation from the proposal distribution q(x);
updating both the first set of parameters representative of adjustable features of the proposal distribution q(x) and the second set of parameters of the base distribution based on the one or more new sample observations such that the proposal distribution q(x) is updated for a next iteration of training;
discarding the new sample observation; and
continuing to alternatively obtain new sample observations and update both the first set of parameters of the proposal distribution q(x) and the second set of parameters of the base distribution until a number of exploration training steps M are taken;
wherein the proposal distribution q(x) following the exploration stage is utilized as an initial proposal distribution q(x) for the exploitation stage; and
wherein T is a total number of training steps and wherein the number of exploitation training steps is T-M.

### Claim 3 (depends on 1)

The computer system of claim 1, wherein the one or more processors are configured for parallel processing using a plurality of processing cores, threads, or processes, and the steps of obtaining the one or more new sample observations and determining the corresponding weight values are conducted on parallel processing cores, threads, or processes of the plurality of processing cores, threads, or processes.

### Claim 4 (depends on 3)

The computer system of claim 3, wherein the iterative steps of obtaining the one or more new sample observations and updating the proposal distribution q(x) are represented as homogenous processing tasks, and the plurality of processing cores, threads, or processes utilize graphics processing units tuned for efficient processing of the homogenous processing tasks.

### Claim 5 (depends on 2)

The computer system of claim 2, wherein a balance between the number of exploration training steps M and the number of exploitation training steps T-M is established using a parameter β, and the parameter β is used to establish an annealing schedule to balance exploration of support of the proposal distribution q(x) and a fine-tuning of regions of the proposal distribution q(x).

### Claim 6 (depends on 1)

The computer system of claim 1, wherein the degree of freedom parameter is adapted for automatically modelling tail-behavior of the target distribution p(x) by adjusting a tail heaviness characteristic of the base distribution, allowing the proposal distribution q(x) to accommodate both light and heavy tailed distribution variants of the target distribution p(x).

### Claim 7 (depends on 1)

The computer system of claim 1, wherein the corresponding weight value is determined using a loss function:
O
x
2
⁡
(
f
j
;
θ
)
=
𝕍
⁡
[
f
j
⁡
(
x
)
⁢
w
]
≈
1
N
⁢
∑
i
⁢
(
f
j
⁡
(
x
(
i
)
)
⁢
p
˜
⁡
(
x
(
i
)
)
q
(
x
(
i
)
)
2
wherein the corresponding weight value is assigned a positive scalar penalty based on a distance from an empirical mean of the proposal distribution q(x), wherein a scalar function ƒj of the loss function is utilized to match a j-th centered moment of the proposal distribution q(x) and the target distribution p(x).

### Claim 8 (depends on 7)

The computer system of claim 7, wherein the scalar function ƒj is defined by:
f
0
⁡
(
x
(
i
)
)
=
1
⁢
⁢
and
⁢
⁢
f
1
⁡
(
x
(
i
)
)
=
1
D
⁢
∑
d
=
1
D
⁢
x
d
(
i
)
⁢
⁢
and
For
⁢
⁢
j
>
1
⁢
f
j
⁡
(
x
(
i
)
)
=
1
D
⁢
∑
d
=
1
D
⁢
(
x
d
(
i
)
-
x
_
d
)
j
where the new sample observations are denoted by x(i) and where the sample mean is defined x=(x1, . . . , xd) where D is an amount of the one or more new sample observations drawn prior to updating the first and the second set of parameters.

### Claim 9 (depends on 1)

The computer system of claim 1, wherein the base distribution is a Student's T-distribution having adjustable tail behavior characteristics using the degree of freedom parameter.

### Claim 10 (depends on 1)

The computer system of claim 1, wherein the first set of parameters are updated based on a previous value of the first set of parameters of a previous iteration.

### Claim 11

A method for generating a Monte Carlo estimation value from a target distribution p(x) using a proposal distribution q(x) representative of a learned importance distribution to encourage computational efficiency of the generation of the Monte Carlo estimation value, the method comprising:
instantiating a proposal distribution model data object having a first set of parameters representative of adjustable features of the proposal distribution q(x);
performing an iterative exploitation training stage for training the proposal distribution model data object, the iterative exploitation training stage including:
iteratively obtaining one or more new sample observations from the proposal distribution q(x) using a normalizing flow model to represent the learned importance distribution, wherein the normalizing flow model is defined using a succession of bi-jective transformations conducted on a random variable z∈D having an base distribution pz(Z) representative of a probability density function of the random variable z, the base distribution having a second set of parameters, including a degree of freedom parameter adapted for modelling tail-behavior of the target distribution p(x);
for each new sample observation of the one or more new sample observations, iteratively determining a corresponding weight value based at least upon a ratio of the target distribution p(x) and the proposal distribution q(x) corresponding to the respective new sample observation;
updating both the first set of parameters representative of adjustable features of the proposal distribution q(x) and the second set of parameters of the base distribution based on the one or more new sample observations such that the proposal distribution q(x) is updated for a next iteration of the exploitation training stage;
recording the one or more new sample observations from the proposal distribution q(x) along with the corresponding weight values; and
continuing to alternatively obtain new sample observations and update both the first set of parameters of the proposal distribution q(x) and the second set of parameters of the base distribution until a number of exploitation training steps are taken; and
generating the Monte Carlo estimation value based at least on a sum of the recorded weight values applied to each of the recorded sample observations; and
storing the Monte Carlo estimation value as a data value in an output data structure.

### Claim 12 (depends on 11)

The method of claim 11, wherein the proposal distribution q(x) is initially poorly fit to the target distribution p(x), and the method further comprises:
performing an iterative exploration training stage prior to the iterative exploitation training stage, the iterative exploration training stage including, iteratively:
iteratively obtaining a new sample observation from the proposal distribution q(x);
updating both the first set of parameters representative of adjustable features of the proposal distribution q(x) and the second set of parameters of the base distribution based on the one or more new sample observations such that the proposal distribution q(x) is updated for a next iteration of training;
discarding the new sample observation; and
continuing to alternatively obtain new sample observations and update both the first set of parameters of the proposal distribution q(x) and the second set of parameters of the base distribution until a number of exploration training steps M are taken;
wherein the proposal distribution q(x) following the exploration stage is utilized as an initial proposal distribution q(x) for the exploitation stage; and
wherein T is a total number of training steps and wherein the number of exploitation training steps is T-M.

### Claim 13 (depends on 11)

The method of claim 11, wherein the method is performed on one or more processors that are configured for parallel processing using a plurality of processing cores, threads, or processes, and the steps of obtaining the one or more new sample observations and determining the corresponding weight values are conducted on parallel processing cores, threads, or processes of the plurality of processing cores, threads, or processes.

### Claim 14 (depends on 13)

The method of claim 13, wherein the iterative steps of obtaining the one or more new sample observations and updating the proposal distribution q(x) are represented as homogenous processing tasks, and the plurality of processing cores, threads, or processes utilize graphics processing units tuned for efficient processing of the homogenous processing tasks.

### Claim 15 (depends on 12)

The method of claim 12, wherein a balance between the number of exploration training steps M and the number of exploitation training steps is established using a parameter β, and the parameter β is used to establish an annealing schedule to balance exploration of the support of the proposal distribution q(x) and a fine-tuning of regions of the proposal distribution q(x).

### Claim 16 (depends on 11)

The method of claim 11, wherein the degree of freedom parameter is adapted for automatically modelling tail-behavior of the target distribution p(x) by adjusting a tail heaviness characteristic of the base distribution, allowing the proposal distribution q(x) to accommodate both light and heavy tailed distribution variants of the target distribution p(x).

### Claim 17 (depends on 11)

The method of claim 11, wherein the corresponding weight value is determined using a loss function:
O
x
2
⁡
(
f
j
;
θ
)
=
𝕍
⁡
[
f
j
⁡
(
x
)
⁢
w
]
≈
1
N
⁢
∑
i
⁢
(
f
j
⁡
(
x
(
i
)
)
⁢
p
˜
⁡
(
x
(
i
)
)
q
(
x
(
i
)
)
2
wherein the corresponding weight value is assigned a based on a distance from an empirical mean of the proposal distribution q(x), wherein a scalar function ƒj of the loss function is utilized to match a j-th centered moment of the proposal distribution q(x) and the target distribution p(x).

### Claim 18 (depends on 17)

The method of claim 17, wherein the scalar function ƒj is defined by:
f
0
⁡
(
x
(
i
)
)
=
1
⁢
⁢
and
⁢
⁢
f
1
⁡
(
x
(
i
)
)
=
1
D
⁢
∑
d
=
1
D
⁢
x
d
(
i
)
⁢
⁢
and
For
⁢
⁢
j
>
1
⁢
f
j
⁡
(
x
(
i
)
)
=
1
D
⁢
∑
d
=
1
D
⁢
(
x
d
(
i
)
-
x
_
d
)
j
where the new sample observations are denoted by x(i) and where the sample mean is defined x=(x1, . . . , xd) where D is an amount of the one or more new sample observations drawn prior to updating the first and the second set of parameters.

### Claim 19 (depends on 11)

The method of claim 11, wherein the base distribution is a Student's T-distribution having adjustable tail behavior characteristics using the degree of freedom parameter

### Claim 20

A non-transitory computer readable medium storing machine interpretable instructions which when executed by one or more processors, cause the one or more processors to perform a method for generating a Monte Carlo estimation value from a target distribution p(x) using a proposal distribution q(x) representative of a learned importance distribution to encourage computational efficiency of the generation of the Monte Carlo estimation value, the method comprising:
instantiating a proposal distribution model data object having a first set of parameters representative of adjustable features of the proposal distribution q(x);
performing an iterative exploitation training stage for training the proposal distribution model data object, the iterative exploitation training stage including:
iteratively obtaining one or more new sample observations from the proposal distribution q(x) using a normalizing flow model to represent the learned importance distribution, wherein the normalizing flow model is defined using a succession of bi-jective transformations conducted on a random variable z∈D having an base distribution pz (Z) representative of a probability density function of the random variable z, the base distribution having a second set of parameters, including a degree of freedom parameter adapted for modelling tail-behavior of the target distribution p(x);
for each new sample observation of the one or more new sample observations, iteratively determining a corresponding weight value based at least upon a ratio of the target distribution p(x) and the proposal distribution q(x) corresponding to the respective new sample observation;
updating both the first set of parameters representative of adjustable features of the proposal distribution q(x) and the second set of parameters of the base distribution based on the one or more new sample observations such that the proposal distribution q(x) is updated for a next iteration of the exploitation training stage;
recording the one or more new sample observations from the proposal distribution q(x) along with the corresponding weight values; and
continuing to alternatively obtain new sample observations and update both the first set of parameters of the proposal distribution q(x) and the second set of parameters of the base distribution until a number of exploitation training steps are taken; and
generating the Monte Carlo estimation value based at least on a sum of the recorded weight values applied to each of the recorded sample observations; and
storing the Monte Carlo estimation value as a data value in an output data structure.
