# US 2021/0342544 A1 · Methods for unsupervised prediction of performance drop due to domain shift

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342544 A1
- Document ID: US20210342544A1
- Status: Published application
- Kind: A1
- Application number: 17/240,216
- Filed: 2021-04-26
- Published: 2021-11-04
- Earliest claimed: 2020-04-30
- Assignees: Naver France
- Applicants: Naver France
- Inventors: Matthias GALLE; Hady ELSAHAR
- Web page: https://patentgrep.com/patent/US20210342544A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 16/3344: {using natural language analysis}
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system includes: a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset; an accuracy module configured to: calculate a domain shift metric based on the input dataset; and calculate a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric; and a retraining module configured to selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.

## Claims

### Claim 1

A system comprising:
a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset;
an accuracy module configured to:
calculate a domain shift metric based on the input dataset; and
calculate a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric; and
a retraining module configured to selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.

### Claim 2 (depends on 1)

The system of claim 1 wherein the retraining module is configured to trigger a retraining of the NLP model when the predicted decrease in accuracy of the NLP model is greater than a predetermined value,
wherein the predetermined value is greater than zero.

### Claim 3 (depends on 1)

The system of claim 1 wherein the domain shift metric is a h-divergence base metric.

### Claim 4 (depends on 3)

The system of claim 3 wherein the accuracy module is configured to calculate the domain shift metric using the equation:
2
⁢
sup
h
∈
H
⁢

Pr
x
~
D
s
⁡
[
h
⁡
(
x
)
=
1
]
-
P
⁢
r
x
~
D
t
⁡
[
h
⁡
(
x
)
⁢
=
1
]

where h is a hypothesis and h(x) the output of that hypothesis on data-point x sampled from one of a training dataset in the training domain and a target dataset in a target domain.

### Claim 5 (depends on 1)

The system of claim 1 wherein the domain shift metric is a confidence metric.

### Claim 6 (depends on 5)

The system of claim 5 wherein the accuracy module is configured to calculate the domain shift metric using the equation:
CONF_CALIB
=
1

U
s

⁢
∑
i
:
x
i
∈
U
s
⁢
-
1

U
t

⁢
∑
j
:
x
j
∈
U
t
⁢
where is a calibrated confidence metric for data-point xi, CONF_CALIB is the domain shift metric, Us represents data in the training domain, and Ut represents data in a target domain.

### Claim 7 (depends on 1)

The system of claim 1 wherein the domain shift metric is a proxy A domain (PAD) metric.

### Claim 8 (depends on 7)

The system of claim 7 wherein the accuracy module is configured to calculate the domain shift metric using the equation
PAD*=1−2ε(G*d(Gf(x))),
where PAD* is the domain shift metric, E is a predetermined scalar value, and θf and θy are learned by minimizing a loss function, and learnable parameters of Gf and G*d are learned parameters.

### Claim 9 (depends on 1)

The system of claim 1 wherein the domain shift metric is a reverse classification accuracy (RCA) metric.

### Claim 10 (depends on 9)

The system of claim 9 wherein the accuracy module is configured to calculate the domain shift metric using the equation:
RCA
*=
1
U
s
′
⁢
∑
x
i
,
y
i
∈
U
s
′
m
′
⁢
⁡
[
y
i
=
C
′
⁡
(
x
i
)
]
-
⁡
[
y
i
=
C
_
⁡
(
x
i
)
]
,
where RCA* is the domain shift metric, C′ is a classifier learned on out-of-domain data and C the classifier learned on held-out in-domain data, xi and yi are data-points, Us′ represents data in the training domain.

### Claim 11 (depends on 1)

The system of claim 1 further comprising a results module configured to:
determine a sentiment regarding a subject based on an output of the NLP model; and
provide results including the sentiment to a computing device from which a query regarding the subject was received.

### Claim 12 (depends on 1)

The system of claim 1 further comprising a results module configured to determine a tag for a point of speech (POS) based on an output of the NLP model.

### Claim 13 (depends on 1)

The system of claim 1, wherein the NLP model is trained based on a training dataset that is different than the input dataset.

### Claim 14 (depends on 13)

The system of claim 13 wherein the training domain of the training dataset is different than a second domain of the input dataset.

### Claim 15 (depends on 1)

The system of claim 1 wherein the input dataset includes text of N social media posts,
wherein N is an integer greater than 0.

### Claim 16

A system comprising:
a training module configured to:
train, using a training dataset in a training domain, a machine learning (ML) model to perform processing on an input dataset;
determine properties of a domain shift metric for the training dataset in the training domain;
an accuracy module configured to:
calculate the domain shift metric based on the input dataset; and
calculate a predicted decrease in accuracy of the ML model attributable to domain shift relative to the training domain based on the domain shift metric using the properties of the domain shift metric for the training dataset; and
a retraining module configured to selectively trigger a retraining of the ML model based on the predicted decrease in accuracy of the ML model.

### Claim 17 (depends on 16)

The system of claim 16 wherein the retraining module is configured to trigger a retraining of the ML model when the predicted decrease in accuracy of the ML model is greater than a predetermined value,
wherein the predetermined value is greater than zero.

### Claim 18 (depends on 16)

The system of claim 16, wherein the ML model is a natural language processing (NLP) model that is trained based on a training dataset that is different than the input dataset.

### Claim 19

A computerized method performed by one or more processors, the method comprising:
training a natural language processing (NLP) model using a training dataset to perform natural language processing on an input dataset;
after receiving the input dataset that is different than the training dataset:
calculating a domain shift metric based on the input dataset; and
calculating a predicted decrease in accuracy of the NLP model attributable to domain shift relative to a training domain based on the domain shift metric; and
selectively triggering a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.

### Claim 20

A system comprising:
a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset;
a means for:
calculating a domain shift metric based on the input dataset; and
calculating a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric; and
a means for selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.
