# US 2024/0202323 A1 · Method and system for protecting a machine learning model against extraction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202323 A1
- Document ID: US20240202323A1
- Status: Published application
- Kind: A1
- Application number: 18/067,043
- Filed: 2022-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: NXP B.V.
- Applicants: NXP B.V.
- Inventors: Jan Hoogerbrugge; Wilhelmus Petrus Adrianus Johannus Michiels
- Web page: https://patentgrep.com/patent/US20240202323A1

## CPC

- G06F 21/554: {involving event detection and direct action}
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 2221/034: Test or assess a computer or a system
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for protecting a Machine Learning (ML) model from extraction have been described. In an illustrative, non-limiting embodiment, a method may include: obtaining a plurality of input samples usable as part of an inference operation, wherein the inference operation is performed through execution of a machine learning (ML) model. The method may further include obtaining a plurality of outputs from the inference operation. The method may further include detecting a temporal inconsistency among at least one of: (a) the plurality of input samples, or (b) the plurality of outputs. Finally, the method may further include identifying an attempt to extract the ML model, based at least in part upon the determination.

## Claims

### Claim 1

A method, comprising:
obtaining a plurality of input samples usable as part of an inference operation, wherein the inference operation is performed through execution of a machine learning (ML) model;
obtaining a plurality of outputs from the inference operation;
detecting a temporal inconsistency among at least one of: (a) the plurality of input samples, or (b) the plurality of outputs; and
based at least in part upon the determination, identifying an attempt to extract the ML model.

### Claim 2 (depends on 1)

The method of claim 1, further comprising performing a responsive action in response to the attempt.

### Claim 3 (depends on 2)

The method of claim 2, wherein the responsive action comprises informing a back-end service via a network connection that the ML model is under an attack.

### Claim 4 (depends on 2)

The method of claim 2, wherein the responsive action comprises reducing an accuracy of the inference operations.

### Claim 5 (depends on 2)

The method of claim 2, wherein the responsive action comprises reducing a speed of the inference operations.

### Claim 6 (depends on 2)

The method of claim 2, wherein the responsive action comprises returning at least one random output for at least one of the plurality of input samples.

### Claim 7 (depends on 6)

The method of claim 6, further comprising reducing a difference between values of consecutive random outputs corresponding to the plurality of input samples.

### Claim 8 (depends on 1)

The method of claim 1, wherein the detecting further comprises:
estimating a motion of groups of pixels from a given input sample to a next successive input sample;
determining a sum of absolute differences (SAD) for at least some of the groups of pixels between the given input sample and the next successive input sample; and
detecting the temporal inconsistency based, at least in part on the SAD being above a threshold.

### Claim 9 (depends on 1)

The method of claim 1, wherein the plurality of outputs comprise a plurality of detected objects of an object detector, and wherein the detecting further comprises:
estimating motion of at least some of the plurality of detected objects from a given output of the plurality of outputs to a next successive output of the plurality of outputs;
determining a sum of absolute differences (SAD) for the at least some of the detected objects between the given output and the next successive output; and
detecting the temporal inconsistency based, at least in part on the SAD being above a threshold.

### Claim 10 (depends on 1)

The method of claim 1, wherein the plurality of outputs from the inference operations of the ML model comprise a plurality of classification results, and wherein the detecting further comprises:
detecting, by a second machine learning model for anomaly detection, at least one anomaly in the plurality of classification results; and
determining that the plurality of classification results are temporally inconsistent based, at least in part, on the detected at least one anomaly.

### Claim 11 (depends on 1)

The method of claim 1, wherein identifying the attempt further comprises determining that the attempt is underway based, at least in part, in response to the temporal inconsistency being above a threshold.

### Claim 12

A system, comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the system to:
deploy a machine learning (ML) model configured to (a) obtain a plurality of successive input samples for inference operations, and (b) provide a plurality of outputs from the inference operations corresponding to the plurality of inputs;
determine that the plurality of input samples or the plurality of outputs are temporally inconsistent; and
based at least in part on the determination of temporal inconsistency, determine that an attempt to copy the ML model is underway.

### Claim 13 (depends on 12)

The system of claim 12, wherein to determine that the plurality of input samples or the plurality of outputs are temporally inconsistent, the program instructions, upon execution by the processor, further cause the system to:
estimate motion of groups of pixels from a given input sample of the plurality of input samples to a next successive input sample of the plurality of input samples;
determine a sum of absolute differences (SAD) for at least some of the groups of pixels between the given input sample and the next successive input sample; and
detect the temporal inconsistency based, at least in part, upon the SAD being greater than a threshold.

### Claim 14 (depends on 12)

The system of claim 12, wherein the plurality of outputs from the inference operations of the ML model comprise a plurality of detected objects of an object detector, and wherein to determine that the plurality of input samples or the plurality of outputs are temporally inconsistent, the program instructions, upon execution by the processor, further cause the system to:
estimate motion of at least some of the plurality of detected objects from a given output of the plurality of outputs to a next successive output of the plurality of outputs;
determine a sum of absolute differences (SAD) for the at least some of the detected objects between the given output and the next successive output; and
detect the temporal inconsistency based, at least in part, upon the SAD being greater than a threshold.

### Claim 15 (depends on 12)

The system of claim 12, wherein the plurality of outputs from the inference operations of the ML model comprise a plurality of classification results, and wherein the program instructions, upon execution by the processor, further cause the system to detect the temporal inconsistency based, at least in part, upon an anomaly in the plurality of classification results, wherein the anomaly is detected using a second ML model.

### Claim 16 (depends on 12)

The system of claim 12, the program instructions, upon execution by the processor, further cause the system to:
inform a back-end service via a network connection that the ML model is under an attack;
lower the accuracy of the inference operations of the ML model;
slow the speed of the inference operations of the ML model; or
return at least one random output for at least one of the plurality of successive input samples.

### Claim 17

A hardware memory having program instructions stored thereon that, upon execution by a computing device, cause the computing device to:
obtain a plurality of successive input samples for inference operations of a machine learning (ML) model;
obtain a plurality of outputs from the inference operations of the ML model corresponding to the plurality of inputs;
determine that either the plurality of input samples or the plurality of outputs are temporally inconsistent; and
based at least in part on the determination of temporal inconsistency, determine that an attempt to copy the ML model is underway.

### Claim 18 (depends on 17)

The hardware memory of claim 17, wherein to determine that either the plurality of input samples or the plurality of outputs are temporally inconsistent, the program instructions further cause the computing device to:
estimate motion of groups of pixels from a given input sample of the plurality of input samples to a next successive input sample of the plurality of input samples;
determine a sum of absolute differences (SAD) for at least some of the groups of pixels between the given input sample and the next successive input sample; and
detect temporal inconsistency based, at least in part, upon the SAD being above a threshold.

### Claim 19 (depends on 17)

The hardware memory of claim 17, wherein the plurality of outputs from the inference operations of the ML model comprise a plurality of detected objects of an object detector, and wherein to determine that either the plurality of input samples or the plurality of outputs are temporally inconsistent, the program instructions further cause the computing device to:
estimate motion of at least some of the plurality of detected objects from a given output of the plurality of outputs to a next successive output of the plurality of outputs;
determine a sum of absolute differences (SAD) for the at least some of the detected objects between the given output and the next successive output; and
detect the temporal inconsistency based, at least in part, upon the SAD being above a threshold.

### Claim 20 (depends on 17)

The hardware memory of claim 17, wherein the plurality of outputs from the inference operations of the ML model comprise a plurality of classification results, and wherein to determine that either the plurality of input samples or the plurality of outputs are temporally inconsistent, the program instructions further cause the computing device to:
detect, by a second ML model, at least one anomaly in the plurality of classification results; and
determine that the plurality of classification results are temporally inconsistent based, at least in part, upon the at least one anomaly.
