# US 12,002,194 B2 · Training an artificial neural network using simulated specimen images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,194 B2 - Document ID: US12002194B2 - Status: Granted - Kind: B2 - Application number: 17/072,035 - Filed: 2020-10-16 - Published: 2021-02-18 - Granted: 2024-06-04 - Earliest claimed: 2018-07-25 - Assignees: FEI Company - Applicants: FEI Company - Inventors: Ond{hacek over (r)}ej Machek; Tomá{hacek over (s)} Vystav{hacek over (e)}l; Libor Strako{hacek over (s)}; Pavel Potocek - Web page: https://patentgrep.com/patent/US12002194B2 ## CPC - G06T 7/001: {using an image reference approach} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06T 2207/10056: Microscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30148: Semiconductor; IC; Wafer - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques for training an artificial neural network (ANN) using simulated specimen images are described. Simulated specimen images are generated based on data models. The data models describe characteristics of a crystalline material and characteristics of one or more defect types. The data models do not include any image data. Simulated specimen images are input as training data into a training algorithm to generate an artificial neural network (ANN) for identifying defects in crystalline materials. After the ANN is trained, the ANN analyzes captured specimen images to identify defects shown therein. ## Claims ### Claim 1 A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware devices, cause performance of operations comprising: obtaining a first data model comprising a description of (a) one or more characteristics of a crystalline material, and (b) one or more characteristics of a first defect type of a structural defect within the crystalline material, wherein the first data model does not include any image data; applying the first data model to a simulation model to generate a first simulated specimen image which shows the crystalline material including a first set of one or more defects of the first defect type; inputting a set of one or more simulated specimen images as training data into a training algorithm to generate an artificial neural network, the set of simulated specimen images including the first simulated specimen image; and generating the artificial neural network, wherein the artificial neural network is configured to detect a second set of one or more defects of the first defect type in the crystalline material as shown in a set of one or more scanned specimen images, wherein the instructions further cause the performance of: receiving an input identifying a modification of the first data model; modifying the first data model based on the input to generate a second data model; applying the second data model to the simulation model to generate a second simulated specimen image; inputting at least the second simulated specimen image as training data into the training algorithm to generate a modified artificial neural network; and generating the modified artificial neural network. ### Claim 2 (depends on 1) The medium of claim 1, wherein the first data model does not comprise data that can be rasterized as a grid of pixels. ### Claim 3 (depends on 1) The medium of claim 1, wherein the instructions further cause the performance of obtaining the first data model. ### Claim 4 (depends on 3) The medium of claim 3, wherein obtaining the first data model comprises receiving one or more user inputs that identify at least one of the characteristics of the crystalline material, and the characteristics of the first defect type. ### Claim 5 (depends on 1) The medium of claim 1, wherein applying the first data model to the simulation model comprises applying the first data model to an image capturing simulator to obtain a raw simulated specimen image. ### Claim 6 (depends on 1) The medium of claim 1, wherein the modification corresponds to a change of one or more of: the characteristics of the crystalline material; the characteristics of the first defect type; microscope characteristics; and noise characteristics. ### Claim 7 (depends on 1) The medium of claim 1, wherein the modified artificial neural network is configured to identify a second defect type in the crystalline material as shown in the set of one or more scanned specimen images. ### Claim 8 (depends on 1) The medium of claim 1, wherein the artificial neural network is configured to identify the first defect type in the crystalline material as shown in the set of one or more scanned specimen images obtained with a first charged particle microscope system having a first set of microscope characteristics, and wherein the modified artificial neural network is configured to identify the first defect type in the crystalline material as shown in a different set of one or more scanned specimen images obtained with a second charged particle microscope system having a second set of microscope characteristics. ### Claim 9 (depends on 1) The medium of claim 1, wherein the first simulated specimen image is a raw simulated specimen image, and the instructions further cause the performance of merging the raw simulated specimen image into an acquired image to generate a final simulated specimen image, wherein the acquired image was obtained with a charged particle microscope system. ### Claim 10 (depends on 9) The medium of claim 9, wherein the set of simulated specimen images including the first simulated specimen image further includes the final simulated specimen image. ### Claim 11 (depends on 9) The medium of claim 9, wherein the first data model further comprises one or more microscopy parameters of a first microscope, the microscopy parameters comprising at least one of: a dwell time; a resolution; an accelerating voltage; a beam current; a convergence angle; and a spot size. ### Claim 12 (depends on 1) The medium of claim 1, wherein the operations further comprise: determining a defect identification rate for the first defect type, wherein obtaining the first data model and applying the first data model to the simulation model are performed responsive to determining that the defect identification rate for the first defect type is lower than a threshold. ### Claim 13 (depends on 1) The medium of claim 1, wherein the first data model comprises a non-image data file. ### Claim 14 (depends on 1) The medium of claim 1, wherein the instructions further cause the performance of: identifying one or more defect identification rates for the artificial neural network that match an improvement needed criteria; identifying one or more correlated characteristics corresponding to the one or more defect identification rates; and generating, based on the one or more correlated characteristics, a second simulated specimen image using the simulation model; and generating a modified artificial neural network using the second simulated specimen image, wherein the modified artificial neural network is configured to have improved defect identification rates for the improvement needed criteria. ### Claim 15 (depends on 14) The medium of claim 14, wherein generating the second simulated specimen image comprises: generating a second data model based on the one or more correlated characteristics, wherein the second data model comprises at least one of (a) a modified characteristic of the crystalline material, and (b) a modified characteristic of the first defect type; and applying the second data model to the simulation model to generate the second simulated specimen image. ### Claim 16 A method, comprising: obtaining a first data model comprising a description of (a) one or more characteristics of a crystalline material, and (b) one or more characteristics of a first defect type of a structural defect within the crystalline material, wherein the first data model does not include any image data; applying the first data model to a simulation model to generate a first simulated specimen image which shows the crystalline material including a first set of one or more defects of the first defect type; inputting a set of one or more simulated specimen images as training data into a training algorithm to generate an artificial neural network, the set of simulated specimen images including the first simulated specimen image; generating the artificial neural network, wherein the artificial neural network is configured to detect a second set of one or more defects of the first defect type in the crystalline material as shown in a set of one or more scanned specimen images; identifying one or more defect identification rates for the artificial neural network that match an improvement needed criteria; identifying one or more correlated characteristics corresponding to the one or more defect identification rates of the artificial neural network; generating, based on the one or more correlated characteristics, a second simulated specimen image using the simulation model; and generating a modified artificial neural network using the second simulated specimen image, wherein the modified artificial neural network is configured to have improved defect identification rates for the improvement needed criteria, and wherein the method is performed using at least one device including a hardware processor. ### Claim 17 (depends on 16) The method of claim 16, wherein generating the second simulated specimen image comprises: generating a second data model based on the one or more correlated characteristics, wherein the second data model comprises at least one of (a) a modified characteristic of the crystalline material, and (b) a modified characteristic of the first defect type; and applying the second data model to the simulation model to generate the second simulated specimen image. ### Claim 18 A system comprising at least one device including a hardware processor, wherein the system is configured to perform operations comprising: obtaining a first data model comprising a description of (a) one or more characteristics of a crystalline material, and (b) one or more characteristics of a first defect type of a structural defect within the crystalline material, wherein the first data model does not include any image data; applying the first data model to a simulation model to generate a first simulated specimen image which shows the crystalline material including a first set of one or more defects of the first defect type; inputting a set of one or more simulated specimen images as training data into a training algorithm to generate an artificial neural network, the set of simulated specimen images including the first simulated specimen image; generating the artificial neural network, wherein the artificial neural network is configured to detect a second set of one or more defects of the first defect type in the crystalline material as shown in a set of one or more scanned specimen images; identifying one or more defect identification rates for the artificial neural network that match an improvement needed criteria; identifying one or more correlated characteristics corresponding to the one or more defect identification rates of the artificial neural network; generating, based on the one or more correlated characteristics, a second simulated specimen image using the simulation model; and generating a modified artificial neural network using the second simulated specimen image, wherein the modified artificial neural network is configured to have improved defect identification rates for the improvement needed criteria.