# US 2021/0174148 A1 · Accuracy of Classification Models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174148 A1
- Document ID: US20210174148A1
- Status: Published application
- Kind: A1
- Application number: 17/107,851
- Filed: 2020-11-30
- Published: 2021-06-10
- Earliest claimed: 2019-12-05
- Assignees: Orbotech Ltd.
- Applicants: Orbotech Ltd.
- Inventors: Gonen Raveh; Elad Goshen
- Web page: https://patentgrep.com/patent/US20210174148A1

## CPC

- G06N 20/00: Machine learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24: Classification techniques
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher}
- G06V 10/82: using neural networks
- G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An Artificial Intelligence system, an apparatus and, a computer program product and a method for automatic improvement of artificial intelligence classification models. A model-performance measurement of the classification model is iteratively improved by at least a predetermined target goal in each iteration. The iterative improvement comprises generating a hypotheses graph for improving the classification model, based on a list of hypotheses and scores thereof. Each hypothesis relates to a strategy for potentially improving the classification model, and is associated with a score indicating a likelihood that an application thereof improves the model-performance measurement. Each node of the hypotheses graph comprises a hypothesis of the list of hypotheses. The iterative improvement further comprises selecting a selected hypothesis from the hypotheses graph based on a traversal thereof; and executing the selected hypothesis thereby updating the classification model and improving the model-performance measurement by at least the predetermined target goal.

## Claims

### Claim 1

A method comprising:
obtaining an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task;
in response to said obtaining, iteratively improving a model-performance measurement of the classification model, wherein in each iteration of the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein said iteratively improving comprises:
determining a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model;
generating, based on the list of hypotheses and the scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses;
selecting a selected hypothesis from the hypotheses graph, wherein said selecting is based on a traversal of the hypotheses graph; and
executing the selected hypothesis thereby updating the classification model and improving the model-performance measurement by at least the predetermined target goal.

### Claim 2 (depends on 1)

The method of claim 1, wherein each hypothesis of the list of hypotheses relates to a modification of at least one of:
a model type of the classification model,
a training parameter of the classification model,
a modeling parameter of the classification model,
a modeling of the supervised image classification task,
a training set utilized to train the classification model,
a validation set utilized to validate the classification model, or
a test set utilized to test the classification model.

### Claim 3 (depends on 2)

The method of claim 2, wherein the hypotheses graph comprises at least one hypothesis that is not related to the model type of the classification model.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determining the list of hypotheses comprises:
applying a predefined best practice hypothesis generator, and
applying a random hypothesis generator.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determining the list of hypotheses comprises applying a Subject Matter Expert (SME) hypothesis generator, wherein the SME hypothesis generator is a computer program product designed by a SME to generate hypotheses based on one or more parameters of the supervised image classification task.

### Claim 6 (depends on 1)

The method of claim 1, wherein said determining the list of hypotheses is performed based on at least one of: a number of classes of the supervised image classification task, a number of images in each class of the supervised image classification task, an image size in the supervised image classification task, or a noisy label estimation measurement of the supervised image classification task.

### Claim 7 (depends on 1)

The method of claim 1, wherein said selecting the selected hypothesis comprises selecting a hypothesis that, when applied, improves the model-performance measurement of the classification task by a highest measurement compared to improvement gained by other hypotheses in the hypotheses graph.

### Claim 8 (depends on 1)

The method of claim 1, wherein said selecting the selected hypothesis comprises selecting a first hypothesis encountered during the traversal of the hypotheses graph that an application thereof improves the model-performance measurement of the classification task by at least the predetermined target goal.

### Claim 9 (depends on 1)

The method of claim 1, wherein said determining the list of hypotheses is performed based on a previous set of hypotheses associated with the supervised classification task.

### Claim 10 (depends on 9)

The method of claim 9, wherein said determining the list of hypotheses comprises applying a reinforcement learning algorithm, wherein the reinforcement learning algorithm utilizes the previous set of hypotheses.

### Claim 11 (depends on 1)

The method of claim 1,
wherein said generating the hypotheses graph comprises: applying a reinforcement learning algorithm, wherein the reinforcement learning algorithm utilizes a sequence of previously generated hypotheses graphs; and
wherein said method further comprises: updating the reinforcement learning algorithm based on said executing.

### Claim 12 (depends on 1)

The method of claim 1, wherein the hypotheses graph comprises a parent node that is connected by one or more directed edges to one or more child nodes, respectively, wherein the parent node represents an abstract hypothesis, wherein each of the one or more child nodes represents a different concretization of the abstract hypothesis, wherein said selecting the selected hypothesis from the hypotheses graph comprises:
in response to a traversal of the parent node, performing an abstract computation relating to the abstract hypothesis; and
in response to traversing one of the one or more child nodes, performing a concrete computation;
wherein the selected hypothesis is selected based on the abstract computation and the concrete computation.

### Claim 13

An Artificial Intelligence (AI) system comprising:
a classification model improvement module configured to obtain an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task;
wherein the classification model improvement module is configured to iteratively improve a model-performance measurement of the classification model, wherein in each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein the classification model improvement module comprises:
a planning module configured to:
determine a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model;
generate, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses;
select a selected hypothesis from the hypotheses graph, wherein said selecting is based on a traversal of the hypotheses graph;
and
an execution module configured to execute the selected hypothesis, thereby updating the classification model and improving the model-performance measurement by at least the predetermined target goal.

### Claim 14 (depends on 13)

The AI system of claim 13, wherein said planning module is configured to:
apply a predefined best practice hypothesis generator to generate at least one hypothesis of the hypotheses list;
apply a random hypothesis generator to generate at least one hypothesis of the hypotheses list; and
apply a Subject Matter Expert (SME) hypothesis generator to generate at least one hypothesis of the hypotheses list, wherein the SME hypothesis generator is a computer program product designed by a SME to generate hypotheses based on one or more parameters of the supervised image classification task.

### Claim 15 (depends on 13)

The AI system of claim 13, wherein said planning module is configured to determine the list of hypotheses based on a previous set of hypotheses associated with the supervised classification task.

### Claim 16 (depends on 15)

The AI system of claim 15, wherein said planning module is configured to determine the list of hypotheses by applying a reinforcement learning algorithm, wherein the reinforcement learning algorithm utilizes the previous set of hypotheses.

### Claim 17 (depends on 13)

The AI system of claim 13, wherein said planning module is configured to:
generating the hypotheses graph by applying a reinforcement learning algorithm, wherein the reinforcement learning algorithm utilizes a sequence of previously generated hypotheses graphs; and
wherein said planning module is configured to update the reinforcement learning algorithm based on execution results of said execution module.

### Claim 18 (depends on 13)

The AI system of claim 13, wherein the hypotheses graph comprises a parent node that is connected by one or more directed edges to one or more child nodes, respectively, wherein the parent node represents an abstract hypothesis, wherein each of the one or more child nodes represents a different concretization of the abstract hypothesis, wherein said planning module is configured to:
in response to a traversal of the parent node, perform an abstract computation relating to the abstract hypothesis; and
in response to traversing one of the one or more child nodes, perform a concrete computation;
wherein the selected hypothesis is selected based on the abstract computation and the concrete computation.

### Claim 19

A computerized apparatus having a processor, the processor being adapted to perform the steps of:
obtaining an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task;
in response to said obtaining, iteratively improving a model-performance measurement of the classification model, wherein in each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein said iteratively improving comprises:
determining a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model;
generating, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses;
selecting a selected hypothesis from the hypotheses graph, wherein said selecting is based on a traversal of the hypotheses graph; and
executing the selected hypothesis thereby updating the classification model and improving the model-performance measurement by at least the predetermined target goal.

### Claim 20

A computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform a method comprising:
obtaining an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task;
in response to said obtaining, iteratively improving a model-performance measurement of the classification model, wherein in each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein said iteratively improving comprises:
determining a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model;
generating, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses;
selecting a selected hypothesis from the hypotheses graph, wherein said selecting is based on a traversal of the hypotheses graph; and
executing the selected hypothesis thereby updating the classification model and improving the model-performance measurement by at least the predetermined target goal.
