# US 2022/0044949 A1 · Interactive and iterative training of a classification algorithm for classifying anomalies in imaging datasets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044949 A1
- Document ID: US20220044949A1
- Status: Published application
- Kind: A1
- Application number: 17/376,664
- Filed: 2021-07-15
- Published: 2022-02-10
- Earliest claimed: 2020-08-06
- Applicants: Carl Zeiss SMT GmbH
- Inventors: Thomas Korb; Philipp Huethwohl; Jens Timo Neumann; Abhilash Srikantha
- Web page: https://patentgrep.com/patent/US20220044949A1

## CPC

- G06F 16/55: Clustering; Classification
- H10P 72/0616: {Monitoring of warpages, curvatures, damages, defects or the like}
- G06F 16/285: {Clustering or classification}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/23: Clustering techniques
- G06F 18/2431: Multiple classes
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 3/096: Transfer learning
- G06N 20/00: Machine learning
- G06T 7/0004: {Industrial image inspection}
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06T 2207/10061: from scanning electron microscope
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20092: Interactive image processing based on input by user
- G06T 2207/30148: Semiconductor; IC; Wafer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H10P: GENERIC PROCESSES OR APPARATUS FOR THE MANUFACTURE OR TREATMENT OF DEVICES COVERED BY CLASS H10
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method includes detecting a plurality of anomalies in an imaging dataset of a wafer. The wafer includes a plurality of semiconductor structures. The method also includes executing multiple iterations. At least some of the iterations include determining a current classification of the plurality of anomalies using a machine-learned classification algorithm and tiles of the imaging dataset associated with the plurality of anomalies. The current classification includes a current set of classes into which the anomalies of the plurality of anomalies are binned. The method further includes, based on at least one decision criterion, selecting at least one anomaly of the plurality of anomalies for a presentation to a user. In addition, the method includes, based on an annotation of the at least one anomaly provided by the user with respect to the current classification, re-training the classification algorithm.

## Claims

### Claim 1

A method, comprising:
detecting a plurality of anomalies in an imaging dataset of a wafer, the wafer comprising a plurality of semiconductor structures; and
executing multiple iterations, at least some iterations of the multiple iterations comprising:
determining a current classification of the plurality of anomalies using a machine-learned classification algorithm and tiles of the imaging dataset associated with the plurality of anomalies, the current classification comprising a current set of classes into which the anomalies of the plurality of anomalies are binned;
based on at least one decision criterion, selecting at least one anomaly of the plurality of anomalies for presentation to a user; and
based on an annotation of the at least one anomaly provided by the user with respect to the current classification, re-training the classification algorithm.

### Claim 2 (depends on 1)

The method of claim 1, wherein the at least one anomaly comprises multiple anomalies, and the at least one decision criterion comprises a similarity measure between the multiple anomalies.

### Claim 3 (depends on 2)

The method of claim 2, further comprising selecting the multiple anomalies to have a high similarity measure between each other.

### Claim 4 (depends on 1)

The method of claim 1, wherein the at least one decision criterion comprises a similarity measure of the selected at least one anomaly and one or more further anomalies that were selected in a previous iteration of the multiple iterations.

### Claim 5 (depends on 4)

The method of claim 4, further comprising selecting the multiple anomalies to have a low similarity measure with respect to the one or more further anomalies that were selected in the previous iteration of the multiple iterations.

### Claim 6 (depends on 1)

The method of any one of claim 1, wherein the at least one anomaly comprises multiple anomalies, and the at least one decision criterion comprises the multiple anomalies being binned into the same class of the current set of classes.

### Claim 7 (depends on 6)

The method of claim 6, wherein the same class comprises at least one of an unknown class or a defect class.

### Claim 8 (depends on 1)

The method of claim 1, wherein the at least one decision criterion comprises the selected at least one anomaly being binned into a predefined class of the set of classes.

### Claim 9 (depends on 1)

The method of claim 1, wherein the at least one decision criterion comprises a population of a class of the set of classes into which the at least one anomaly is binned.

### Claim 10 (depends on 1)

The method of any one of claim 1, wherein the at least one decision criterion comprises a context of the selected at least one anomaly with respect to the semiconductor structures.

### Claim 11 (depends on 1)

The method of claim 1, wherein the at least one decision criterion implements at least one member selected from the group consisting of an explorative annotation scheme and an exploitative annotation scheme.

### Claim 12 (depends on 1)

The method of claim 1, wherein the at least one decision criterion differs for at least two iterations of the at least some iterations.

### Claim 13 (depends on 1)

The method of claim 1, wherein an aggregated count of the anomalies selected for presentation to the user across the multiple iterations is at most 50% of a count of the plurality of iterations.

### Claim 14 (depends on 1)

The method of claim 1, wherein the annotation of the at least one anomaly comprises a new class to be added to the current set of classes.

### Claim 15 (depends on 1)

The method of claim 1, further comprising, in a first iteration of the multiple iterations, performing an unsupervised clustering of the plurality of anomalies,
wherein the at least one anomaly is selected based on the unsupervised clustering.

### Claim 16 (depends on 1)

The method of claim 1, further comprising aborting execution of the multiple iterations based on at least one abort criterion,
wherein the abort criterion is selected from the group consisting of a user input, a number of classes for which anomalies have been presented to the user, a population of classes in the current set of classes, a probability of finding a new class not yet included in the set of classes, a worst classification confidence of all un-annotated anomalies, and an aggregated count of anomalies selected for presentation to the user or annotated by the user reaching a threshold.

### Claim 17 (depends on 1)

The method of claim 1, wherein the at least one anomaly comprises multiple anomalies concurrently presented to the user, the method further comprises using a user interface to present to the user, and the user interface is configured to batch annotate the multiple anomalies.

### Claim 18 (depends on 17)

The method of claim 17, wherein batch annotation of the multiple anomalies comprises batch assigning of a plurality of labels to the multiple anomalies concurrently presented to the user.

### Claim 19 (depends on 1)

The method of claim 1, wherein the at least one anomaly comprises multiple anomalies concurrently presented to the user, and the method further comprises grouping and/or sorting the multiple anomalies to present to the user.

### Claim 20 (depends on 1)

The method of claim 1, wherein, for a first iteration of the multiple iterations, the machine-learned classification algorithm is pre-trained based on: i) an imaging dataset of a further wafer comprising further semiconductor structures sharing one or more features with semiconductor structures of the plurality of semiconductor structures; or ii) a preclassification using a further classification algorithm.

### Claim 21 (depends on 1)

The method of claim 1, further comprising one of the following:
detecting the plurality of anomalies using an autoencoder neural network and based on a comparison between an input tile of the imaging data provided to the autoencoder neural network and a reconstructed representation of the input tile output by the autoencoder neural network; and
detecting the plurality of anomalies using a die-to-die and/or die-to-database registration.

### Claim 22 (depends on 1)

The method of claim 1, wherein the tiles of the imaging data comprise the anomalies and a surrounding of the anomalies.

### Claim 23 (depends on 1)

The method of claim 1, wherein the current set of classes comprises at least one defect class and at least one nuisance class.

### Claim 24 (depends on 1)

The method of claim 1, further comprising determining a defect density for multiple regions of the wafer based on the machine-learned classification algorithm and the plurality of anomalies, wherein different ones of the multiple regions are associated with different process parameters of a manufacturing process of the semiconductor structures.

### Claim 25 (depends on 1)

The method of claim 1, wherein the imaging dataset is a multibeam SEM image.

### Claim 26 (depends on 1)

The method of claim 1, wherein detecting the plurality of anomalies and the executing of the multiple iterations is part of a work-flow comprising a sequence of:
preconditioning the imaging dataset;
detecting of the plurality of anomalies;
executing of the plurality of iterations;
basing one or more measurements on the classification; and
visualizing and/or reporting.

### Claim 27 (depends on 1)

One or more machine-readable hardware storage devices comprising instructions that are executable by one or more processing devices to perform operations comprising the method of claim 1.

### Claim 28 (depends on 1)

A system comprising:
one or more processing devices; and
one or more machine-readable hardware storage devices comprising instructions that are executable by the one or more processing devices to perform operations comprising the method of claim 1.