# US 2020/0401503 A1 · System and Method for Testing Artificial Intelligence Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401503 A1
- Document ID: US20200401503A1
- Status: Published application
- Kind: A1
- Application number: 16/750,890
- Filed: 2020-01-23
- Published: 2020-12-24
- Earliest claimed: 2019-06-24
- Assignees: Zeyu GAO
- Inventors: Zeyu GAO
- Web page: https://patentgrep.com/patent/US20200401503A1

## CPC

- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F 11/3672: {Test management}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3692: {for test results analysis}
- G06F 11/3696: {Methods or tools to render software testable}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method and a system are provided of testing AI applications/systems on a System for testing one or more artificial intelligence System (STAIS) connectable to under-test AI systems (UTAIS) via the AI test connectors/interfaces through AI test platform. The method includes test modeling, data preparation, testing script generation, testing automation, and quality assurance. The system automatically identifies, analyzes, and displays all quality issues of UTAIS.

## Claims

### Claim 1

A System for testing an artificial intelligence (AI) System(s) (STAIS), wherein the artificial intelligence system(s), to be tested, is/are herein distinctively and specifically described as an Under-test artificial intelligence System(s) (UTAIS), comprises
a. an AI test framework;
b. an AI test platform;
c. an AI test resource library; and
d. an AI test solution.

### Claim 2 (depends on 1)

The system of claim 1 wherein the AI test framework comprises
a. a program supporting and performing AI test analysis and modeling;
b. a program supporting and performing AI test quality evaluation;
c. a set of AI test connectors and interfaces; and
d. a program generating AI test scripts.

### Claim 3 (depends on 1)

The system of claim 1 wherein the AI test platform comprises
a. a sub-platform for testing medical/healthcare application specific UTAIS;
b. a sub-platform for testing robotics application specific UTAIS;
c. a sub-platform for testing mobile application specific UTAIS;
d. a sub-platform for testing unmanned vehicle/drone application specific UTAIS; and
e. a sub-platform for testing business application specific UTAIS.

### Claim 4 (depends on 1)

The system of claim 1 wherein the AI test resource library comprises
a. a sub-library of classified background media including classified background images, videos, and audios;
b. a sub-library of classified object media including classified object images, videos, and audios;
c. a sub-library of classified texts and strings;
d. a sub-library of classified data augmentation; and
e. a sub-library of classified data models.

### Claim 5 (depends on 1)

The system of claim 1 wherein the AI test solution comprises
a. one or more AI-based test modeling engine(s)/program(s) using AI techniques for discovering existing AI function test models and assisting derivation of new AI function test models;
b. one or more AI-based test scripting engine(s)/program(s) using AI techniques for assisting new script derivation;
c. one or more AI-based test case engine(s)/program(s) using AI techniques for selecting test cases;
d. one or more AI-based debugging engine(s)/program(s) using AI techniques for detecting bugs and analyzing detected bugs; and
e. one or more AI-based test data recommendation engine(s)/program(s) using AI techniques for recommending test data.

### Claim 6 (depends on 2)

The system of claim 2 wherein the program supporting and performing AI test analysis and modeling comprises
a. a sub-program for AI requirement analysis modeling;
b. a sub-program for AI function test modeling;
c. a sub-program for multi-dimension (mD) classification;
d. a sub-program for white-box AI test modeling; and
e. a sub-program for test data modeling.

### Claim 7 (depends on 2)

The system of claim 2 wherein the program generating AI test scripts comprises
a. an AI test generator supporting multi-dimension (mD) test generation;
b. an AI test specific scripting language as an extension of existing test script language;
c. an AI test scripting program supporting test script generation; and
d. an AI test runner supporting auto test execution, control, and monitoring.

### Claim 8 (depends on 2)

The system of claim 2 wherein the set of AI test connectors or interfaces comprises
a. a set of AI test connectors or interfaces supporting test automation and service for AI-powered medical applications, i.e., medical application specific artificial intelligence system(s);
b. a set of AI test connectors or interfaces supporting test automation and service for AI-powered robotics applications, i.e., robotics application specific artificial intelligence system(s);
c. a set of AI test connectors or interfaces supporting test automation and service for AI-powered unmanned vehicle applications, i.e., unmanned vehicle application specific artificial intelligence system(s);
d. a set of AI test connectors or interfaces supporting test automation and service for AI-powered business applications, i.e., business application specific artificial intelligence system(s); and
e. a set of AI test connectors or interfaces supporting test automation and service for AI-powered mobile applications, i.e., mobile application specific artificial intelligence system(s).

### Claim 9 (depends on 2)

The system of claim 2 wherein the program performing AI test quality evaluation comprises
a. a sub-program supporting an AI function quality assessment;
b. a sub-program supporting an AI system quality assessment;
c. a sub-program having requirement-based AI test coverages;
d. a sub-program supporting model-based AI test coverage analyses;
e. a sub-program supporting AI function test complexity analyses; and
f. a sub-program supporting AI system test complexity analyses.

### Claim 10 (depends on 3)

The system of claim 3 wherein the sub-platform for testing medical/healthcare application specific UTAIS comprises
a. an intelligent medical system test simulator interfaceable with medical/healthcare application specific UTAIS; and
b. an intelligent medical system test service manager interfaceable with the intelligent medical system test simulator and the medical/healthcare application specific UTAIS.

### Claim 11 (depends on 3)

The system of claim 3 wherein the sub-platform for testing robotics application specific UTAIS comprises
a. an intelligent robot test simulator interfaceable with robotics application specific UTAIS; and
b. an intelligent robot test service manager interfaceable with the intelligent robot test simulator and the robotics application specific UTAIS.

### Claim 12 (depends on 3)

The system of claim 3 wherein the sub-platform for mobile application specific UTAIS comprises
a. an intelligent mobile system test simulator interfaceable with mobile application specific UTAIS; and
b. an intelligent mobile system test service manager interfaceable with the intelligent mobile system test simulator and the mobile application specific UTAIS.

### Claim 13 (depends on 3)

The system of claim 3 wherein the sub-platform for unmanned vehicle applications specific UTAIS comprises
a. an intelligent unmanned vehicle/drone test simulator interfaceable with unmanned vehicle application specific UTAIS; and
b. an intelligent unmanned vehicle/drone test service manager-with the intelligent unmanned vehicle/drone test simulator and the unmanned vehicle application specific UTAIS.

### Claim 14 (depends on 3)

The system of claim 3 wherein the sub-platform for business application specific UTAIS comprises
a. an intelligent business system test simulator interfaceable with business application specific UTAIS; and
b. an intelligent business system test service manager interfaced with the intelligent business system test simulator and the business application specific UTAIS.

### Claim 15 (depends on 5)

The system of claim 5 wherein the AI-based test modeling engine(s)/program(s) using AI techniques for discovering AI function test models comprises
a. a user assisted AI test modeling program(s);
b. an AI-based test model classification program(s);
c. an AI-based test model comparison program(s);
d. an AI-based test model recommendation program(s); and
e. an AI-based test model quality assessment program(s).

### Claim 16 (depends on 5)

The system of claim 5 wherein the set of AI-based test scripting engine(s)/program(s) using AI techniques for generating scripts comprises
a. an AI-based test script search and selection;
b. an AI-based test script classification;
c. an AI-based test script generation;
d. an AI-based test script recommendation; and
e. an AI-based test script quality assessment.

### Claim 17 (depends on 5)

The system of claim 5 wherein the AI-based test case engine/program using AI techniques for selecting test cases comprises
a. an AI-based test case search and selection;
b. an AI-based test case classification;
c. an AI-based test case generation;
d. an AI-based test case recommendation; and
e. an AI-based test case quality assessment.

### Claim 18 (depends on 5)

The system of claim 5 wherein the AI-based debugging engine/program using AI techniques for detecting bugs and analyzing detected bugs comprises
a. an AI-based bug report;
b. an AI-based bug classification;
c. an AI-based debugging generation;
d. an AI-based debugging validation; and
e. an AI-based debugging quality assessment.

### Claim 19 (depends on 5)

The system of claim 5 wherein the AI-based test data recommendation engine/program using AI techniques for recommending test data comprises
a. an AI-based big data search;
b. an AI-based test data classification;
c. an AI-based data generation and augmentation;
d. an AI-based test data validation; and
e. an AI-based test data quality assessment.

### Claim 20

A method for testing an artificial intelligence system(s), wherein the artificial intelligence system(s) is/are herein distinctively and specifically described as under-test artificial intelligence system(s) (UTAIS), comprises
a. having an AI test framework;
b. having an AI test platform;
c. having an AI test resource library; and
d. having an AI test solution(s).

### Claim 21 (depends on 20)

The method of claim 20 wherein having an AI test framework comprises
a. having a program performing AI test analysis and modeling;
b. having a program performing AI test quality evaluation;
c. having a set of AI test connectors and interfaces; and
d. having a program generating AI test scripts.

### Claim 22 (depends on 20)

The method of claim 20 wherein having an AI test platform comprises
a. having a sub-platform for testing a medical/healthcare application specific UTAIS;
b. having a sub-platform for testing a robotics application specific UTAIS;
c. having a sub-platform for testing a mobile application specific UTAIS;
d. having a sub-platform for testing an unmanned vehicle application specific UTAIS; and
e. having a sub-platform for testing a business application specific UTAIS.

### Claim 23 (depends on 20)

The method of claim 20 wherein having an AI test resource library comprises
a. having a sub-library of classified background media including images, videos, and audios;
b. having a sub-library of classified object media including images, videos, and audios;
c. having a sub-library of classified texts and strings;
d. having a sub-library of classified data augmentation; and
e. having a sub-library of classified data models.

### Claim 24 (depends on 20)

The method of claim 20 wherein having an AI test solution(s) comprises
a. having an AI-based test modeling engine/program using AI techniques for discovering AI function test models;
b. having an AI-based test scripting engine/program using AI techniques for generating scripts;
c. having an AI-based test case engine/program using AI techniques for selecting test cases;
d. having an AI-based debugging engine/program using AI techniques for detecting bugs and analyzing detected bugs; and
e. having an AI-based test data recommendation engine/program using AI techniques for recommending test data.

### Claim 25 (depends on 21)

The method of claim 21 wherein having a program performing AI test analysis and modeling comprises
a. having a sub-program for AI requirement analysis modeling;
b. having a sub-program for AI function test modeling;
c. having a sub-program having a multi-dimensional (mD) classification;
d. having a sub-program for white-box AI test modeling; and
e. having a sub-program for test data modeling.

### Claim 26 (depends on 21)

The method of claim 21 wherein having a program generating AI test scripts comprises
a. having an AI test generator;
b. having an AI test language;
c. having an AI test scripting program; and
d. having an AI test runner.

### Claim 27 (depends on 21)

The system of claim 21 wherein having a set of AI test connectors or interfaces comprises
a. having a set of AI test connectors or interfaces supporting test automation and service for AI-powered medical applications, i.e., medical application specific artificial intelligence system(s);
b. having a set of AI test connectors or interfaces supporting test automation and service for AI-powered robotics applications, i.e., robotics application specific artificial intelligence system(s);
c. having a set of AI test connectors or interfaces supporting test automation and service for AI-powered unmanned vehicle applications, i.e., unmanned vehicle application specific artificial intelligence system(s);
d. having a set of AI test connectors or interfaces supporting test automation and service for AI-powered business applications, i.e., business application specific artificial intelligence system(s); and
e. having a set of AI test connectors or interfaces supporting test automation and service for AI-powered mobile applications, i.e., mobile application specific artificial intelligence system(s).

### Claim 28 (depends on 21)

The method of claim 21 wherein having a program performing AI test quality evaluation comprises
a. having a sub-program for an AI function quality assessment;
b. having a sub-program for an AI system quality assessment;
c. having a sub-program having requirement-based AI test coverages;
d. having a sub-program for model-based AI test coverage analyses;
e. having a sub-program for AI function test complexity analyses; and
f. having a sub-program for AI system test complexity analyses.

### Claim 29 (depends on 22)

The system of claim 22 wherein having a sub-platform for testing medical/healthcare application specific UTAIS comprises
a. having an intelligent medical system test simulator interfaceable with medical/healthcare application specific UTAIS and
b. having an intelligent medical system test service manager interfaceable with the intelligent medical system test simulator and the medical/healthcare application specific UTAIS.

### Claim 30 (depends on 22)

The system of claim 22 wherein having a sub-platform for testing robotics application specific UTAIS comprises
a. having an intelligent robot test simulator interfaceable with robotics application specific UTAIS; and
b. having an intelligent robot test service manager interfaceable with the intelligent robot test simulator and the robotics application specific UTAIS.

### Claim 31 (depends on 22)

The system of claim 22 wherein having a sub-platform for mobile application specific UTAIS comprises
a. having an intelligent mobile system test simulator interfaceable with mobile application specific UTAIS; and
b. having an intelligent mobile system test service manager interfaceable with the intelligent mobile system test simulator and the mobile application specific UTAIS.

### Claim 32 (depends on 22)

The system of claim 22 wherein having a sub-platform for unmanned vehicle application specific UTAIS comprises
a. having an intelligent unmanned vehicle/drone test simulator interfaceable with unmanned vehicle application specific UTAIS; and
b. having an intelligent unmanned vehicle/drone test service manager interfaceable with the intelligent unmanned vehicle/drone test simulator and the unmanned vehicle application specific UTAIS.

### Claim 33 (depends on 22)

The system of claim 22 wherein having a sub-platform for business applications specific UTAIS comprises
a. having an intelligent business system test simulator interfaceable with business application specific UTAIS; and
b. having an intelligent business system test service manager interfaced with the intelligent business system test simulator and the business application specific UTAIS.

### Claim 34 (depends on 24)

The method of claim 24 wherein having an AI-based test modeling engine/program using AI techniques for discovering AI function test models comprises
a. having a user assisted AI test modeling;
b. having an AI-based test model classification;
c. having an AI-based test model comparison;
d. having an AI-based test model recommendation; and
e. having an AI-based test model quality assessment.

### Claim 35 (depends on 24)

The method of claim 24 wherein having an AI-based test scripting engine/program using AI techniques for generating scripts comprises
a. having an AI-based test script search and selection;
b. having an AI-based test script classification;
c. having an AI-based test script generation;
d. having an AI-based test script recommendation; and
e. having an AI-based test script quality assessment.

### Claim 36 (depends on 24)

The method of claim 24 wherein having an AI-based test case engine/program using AI techniques for selecting test cases comprises
a. having an AI-based test case search and selection;
b. having an AI-based test case classification;
c. having an AI-based test case generation;
d. having an AI-based test case recommendation; and
e. having an AI-based test case quality assessment.

### Claim 37 (depends on 24)

The method of claim 24 wherein having an AI-based debugging engine/program using AI techniques for detecting bugs and analyzing detected bugs comprises
a. having an AI-based bug report;
b. having an AI-based bug classification;
c. having an AI-based debugging generation;
d. having an AI-based debugging validation; and
e. having an AI-based debugging quality assessment.

### Claim 38 (depends on 24)

The method of claim 24 wherein having an AI-based test data recommendation engine/program using AI techniques for recommending test data comprises
a. having an AI-based big data search;
b. having an AI-based test data classification;
c. having an AI-based data generation and augmentation;
d. having an AI-based test data validation; and
e. having an AI-based test data quality assessment.