# US 2021/0279336 A1 · Method for testing an artificial intelligence model using a substitute model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279336 A1
- Document ID: US20210279336A1
- Status: Published application
- Kind: A1
- Application number: 16/808,982
- Filed: 2020-03-04
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Lukasz G. Cmielowski; Rafal Bigaj; Wojciech Sobala; Maksymilian Erazmus
- Web page: https://patentgrep.com/patent/US20210279336A1

## CPC

- G06F 21/577: {Assessing vulnerabilities and evaluating computer system security}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F 2221/033: Test or assess software
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure relates to a computer implemented method for testing an artificial intelligence module (AI-module). The method comprises generating a substitute module using first input datasets and first output datasets, wherein the first output datasets are generated on the basis of the first input datasets using the AI-module. Adversarial input datasets are generated on the basis of the first input datasets using the substitute module. The adversarial input datasets are used for assessing a resilience of the AI-module against adversarial attacking by using the first output datasets and second output datasets, wherein the second output datasets are generated on the basis of the adversarial input datasets using the AI-module.

## Claims

### Claim 1

A computer implemented method for testing an artificial intelligence module (AI-module), the method comprising:
sending first input datasets to an interface of the AI-module for processing the first input datasets by the AI-module;
receiving first output datasets calculated by the AI-module on the basis of the first input datasets via the interface of the AI-module, each first output dataset corresponding to a respective one of the first input datasets;
generating a substitute module using the first input datasets and the first output datasets, the substitute module substituting the AI-module;
generating adversarial input datasets on the basis of the first input datasets using the substitute module;
sending the adversarial input datasets to the interface of the AI-module for processing the adversarial input datasets by the AI-module;
receiving second output datasets calculated by the AI-module on the basis of the adversarial input datasets via the interface of the AI-module, each second output dataset corresponding to a respective one of the adversarial input datasets;
assessing a resilience of the AI-module against adversarial attacking by using the first output datasets and the second output datasets, wherein sending the adversarial input datasets to the interface of the AI-module represents the adversarial attacking.

### Claim 2 (depends on 1)

The method of claim 1, the method further comprising calculating a metric for assessing the resilience of the AI-module against adversarial attacking.

### Claim 3 (depends on 2)

The method of claim 2, the method further comprising calculating a first score as the metric, the first score representing a proportion of successful adversarial attacks from all adversarial attacks using the adversarial input datasets.

### Claim 4 (depends on 1)

The method of claim 1, the method further comprising generating the adversarial input datasets according to a spatial transformation attack method.

### Claim 5 (depends on 1)

The method of claim 1, the method further comprising generating the adversarial input datasets according to a zeroth order oracle attack method.

### Claim 6 (depends on 1)

The method of claim 1, the method further comprising generating the adversarial input datasets according to a decision based attack method.

### Claim 7 (depends on 1)

The method of claim 1, the method further comprising generating the adversarial input datasets according to a query-efficient black box attack method.

### Claim 8 (depends on 1)

The method of claim 1, the method further comprising generating the adversarial input datasets according to a fast gradient method.

### Claim 9 (depends on 1)

The method of claim 1, the method further comprising generating a set of potential substitute modules with different module architectures using the first input datasets and the first output datasets, each potential substitute module substituting the AI-module, performing a comparison of performances of the potential substitute modules with respect to how accurately the potential substitute modules imitate the AI-module and selecting the substitute module from one of the potential substitute modules on the basis of the comparison.

### Claim 10 (depends on 9)

The method of claim 9, the method further comprising performing the comparison of the performances of the potential substitute modules with respect to how accurately the potential substitute modules imitate the AI-module on the basis of a second metric.

### Claim 11 (depends on 10)

The method of claim 10, wherein the second metric is a Cohen's kappa coefficient.

### Claim 12 (depends on 1)

The method of claim 1, the method further comprising
constructing non-adversarial input datasets and determining non-adversarial output datasets on the basis of the non-adversarial input datasets using the substitute module, each non-adversarial output dataset corresponding to a respective one of the non-adversarial input datasets;
sending the non-adversarial input datasets to the interface of the AI-module for processing the non-adversarial input datasets by the AI-module;
receiving third output datasets calculated by the AI-module on the basis of the non-adversarial input datasets via the interface of the AI-module, each third output dataset corresponding to a respective one of the non-adversarial input datasets;
assessing the resilience of the AI-module against adversarial attacking in relation to non-adversarial use cases of the AI-module using the non-adversarial output datasets and the third output datasets, the non-adversarial use cases comprising an application of the AI-module to the non-adversarial input datasets.

### Claim 13 (depends on 12)

The method of claim 12, the method further comprising calculating a single third metric for assessing the resilience of the AI-module against adversarial attacking in relation to the non-adversarial use cases of the AI-module.

### Claim 14 (depends on 13)

The method of claim 13, the method further comprising calculating a second score as the third metric, the second score representing a ratio of a number of true positive instances to a sum of a number of the true positive instances and a number of false positive instances, the true positive instances representing cases, for which the adversarial attacking is successful, the false positive instances representing cases, for which the non-adversarial output datasets deviate from the third output datasets. (the latter cases may happen as the substitute module may be slightly different to the AI-module. If I am wrong, please let me know.)

### Claim 15 (depends on 1)

The method of claim 1, the method further comprising training the AI-module using the adversarial input datasets and the first output datasets.

### Claim 16 (depends on 1)

The method of claim 1, the method further comprising training a classifier for classifying an adversarial attack to the AI-module using the adversarial input datasets and the first input datasets.

### Claim 17 (depends on 16)

The method of claim 16, the method further comprising using the classifier for assessing a risk of the AI-module being under adversarial attack with respect to a batch of further input datasets, the further input datasets comprising further adversarial input datasets for the AI-module and further non-adversarial input datasets for the AI-module, calculating a fourth metric on the basis of the batch of the further input datasets using the classifier, the fourth metric representing the risk of the AI-module being under adversarial attack.

### Claim 18 (depends on 17)

The method of claim 17, the fourth metric representing a ratio of a number of detected adversarial attacks to the AI-module being detected by the classifier to a number of detected non-adversarial use cases of the AI-module being detected by the classifier.

### Claim 19

A computer program product comprising a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to implement a method of:
sending first input datasets to an interface of the AI-module for processing the first input datasets by the AI-module;
receiving first output datasets calculated by the AI-module on the basis of the first input datasets via the interface of the AI-module, each first output dataset corresponding to a respective one of the first input datasets;
generating a substitute module using the first input datasets and the first output datasets, the substitute module substituting the AI-module;
generating adversarial input datasets on the basis of the first input datasets using the substitute module;
sending the adversarial input datasets to the interface of the AI-module for processing the adversarial input datasets by the AI-module;
receiving second output datasets calculated by the AI-module on the basis of the adversarial input datasets via the interface of the AI-module, each second output dataset corresponding to a respective one of the adversarial input datasets;
assessing a resilience of the AI-module against adversarial attacking by using the first output datasets and the second output datasets, wherein sending the adversarial input datasets to the interface of the AI-module represents the adversarial attacking.

### Claim 20

A computer system for testing an artificial intelligence module (AI-module), the computer system comprising:
at least one processor;
at least one memory component;
the at least one processor, and the at least one memory component configured to execute a method of:
sending first input datasets to an interface of the AI-module for processing the first input datasets by the AI-module;
receiving first output datasets calculated by the AI-module on the basis of the first input datasets via the interface of the AI-module, each first output dataset corresponding to a respective one of the first input datasets;
generating a substitute module using the first input datasets and the first output datasets, the substitute module substituting the AI-module;
generating adversarial input datasets on the basis of the first input datasets using the substitute module;
sending the adversarial input datasets to the interface of the AI-module for processing the adversarial input datasets by the AI-module;
receiving second output datasets calculated by the AI-module on the basis of the adversarial input datasets via the interface of the AI-module, each second output dataset corresponding to a respective one of the adversarial input datasets;
assessing a resilience of the AI-module against adversarial attacking by using the first output datasets and the second output datasets, wherein sending the adversarial input datasets to the interface of the AI-module represents the adversarial attacking.
