# US 2021/0279520 A1 · Defect Detection System > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279520 A1 - Document ID: US20210279520A1 - Status: Published application - Kind: A1 - Application number: 17/195,760 - Filed: 2021-03-09 - Published: 2021-09-09 - Earliest claimed: 2020-03-09 - Assignees: Nanotronics Imaging, Inc. - Applicants: Nanotronics Imaging, Inc. - Inventors: Tonislav Ivanov; Denis Babeshko; Vadim Pinskiy; Matthew C. Putman; Andrew Sundstrom - Web page: https://patentgrep.com/patent/US20210279520A1 ## CPC - G06V 10/82: using neural networks - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/41: {Interactive pattern learning with a human teacher} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/091: Active learning - G06N 20/20: Ensemble learning - G06T 7/001: {using an image reference approach} - G06V 10/235: {based on user input or interaction} - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/7747: {Organisation of the process, e.g. bagging or boosting} - G06V 10/7784: {based on feedback from supervisors} - G06V 10/7796: {based on specific statistical tests} - G06V 10/945: {User interactive design; Environments; Toolboxes} - G06T 2207/30148: Semiconductor; IC; Wafer - G06V 2201/06: Recognition of objects for industrial automation - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computing system generates a training data set for training the prediction model to detect defects present in a target surface of a target specimen and training the prediction model to detect defects present in the target surface of the target specimen based on the training data set. The computing system generates the training data set by identifying a set of images for training the prediction model, the set of images comprising a first subset of images. A deep learning network generates a second subset of images for subsequent labelling based on the set of images comprising the first subset of images. The deep learning network generates a third subset of images for labelling based on the set of images comprising the first subset of images and the labeled second subset of images. The computing system continues the process until a threshold number of labeled images is generated. ## Claims ### Claim 1 A computing system comprising: a processor; and a memory having programming instructions stored thereon, which, when executed by the processor, performs operations comprising: generating a training data set for training the prediction model to detect defects present in a target surface of a target specimen by: identifying a set of images for training the prediction model, the set of images comprising a first subset of images, wherein each image of the first subset of images is labeled with labels identifying defects on a respective specimen; generating, by a deep learning network, a second subset of images for subsequent labelling based on the set of images comprising the first subset of images; prompting an operator to label each image in the second subset of images; generating, by the deep learning network, a third subset of images for labelling based on the set of images comprising the first subset of images and the labeled second subset of images; prompting the operator to label each image in the third subset of images; and aggregating the first subset of images, the second subset of images, and the third subset of images, wherein each image in the first subset of images, the second subset of images, and the third subset of images are labeled; and training the prediction model to detect defects present in the target surface of the target specimen based on the training data set. ### Claim 2 (depends on 1) The computing system of claim 1, wherein generating, by the deep learning network, the second subset of images for subsequent labelling based on the set of images comprising the first subset of images comprises: for each image in the set of images, generating a first probability map corresponding to a first probability of each pixel in the image belongs to a first category; and for each image in the set of images, generating a second probability map corresponding to a second probability of each pixel in the image belongs to a second category. ### Claim 3 (depends on 2) The computing system of claim 2, further comprising: for each image in the set of images, generating an unseen metric based on the first probability map and the second probability map, wherein the unseen metric corresponds to a likelihood of the image including pixels unseen by the deep learning network. ### Claim 4 (depends on 3) The computing system of claim 3, wherein the unseen metric for the image is based on an aggregate of unseen metrics for each pixel in the image. ### Claim 5 (depends on 4) The computing system of claim 4, further comprising: generating an alpha metric for the image based a corresponding unseen metric. ### Claim 6 (depends on 4) The computing system of claim 4, further comprising: generating a threshold metric for the image based on a number of pixels in the image having an unseen score that exceeds a threshold value. ### Claim 7 (depends on 1) The computing system of claim 1, further comprising: receiving, by the computing system from an imaging apparatus, a target image of the target surface of the target specimen; detecting, by the prediction model, one or more defects present in the target surface of the target specimen; and based on the detecting, generating, by the computing system, a graphical output illustrating the one or more defects. ### Claim 8 A method, comprising: identifying, by a computing system, a set of images for training a prediction model to detect defects in a surface of a specimen, the set of images comprising a first subset of images, wherein each image of the first subset of images is labeled with labels identifying defects on a respective specimen; generating, by a deep learning network of the computing system, a second subset of images for subsequent labelling based on the set of images comprising the first subset of images; prompting, by the computing system, an operator to label each image in the second subset of images; generating, by the deep learning network, a third subset of images for labelling based on the set of images comprising the first subset of images and the labeled second subset of images; prompting, by the computing system, the operator to label each image in the third subset of images; and aggregating, by the computing system, the first subset of images, the second subset of images, and the third subset of images, wherein each image in the first subset of images, the second subset of images, and the third subset of images are labeled to form a training data set for training the prediction model. ### Claim 9 (depends on 8) The method of claim 8, wherein generating, by the deep learning network, the second subset of images for subsequent labelling based on the set of images comprising the first subset of images comprises: for each image in the set of images, generating a first probability map corresponding to a first probability of each pixel in the image belongs to a first category; and for each image in the set of images, generating a second probability map corresponding to a second probability of each pixel in the image belongs to a second category. ### Claim 10 (depends on 9) The method of claim 9, further comprising: for each image in the set of images, generating an unseen metric based on the first probability map and the second probability map, wherein the unseen metric corresponds to a likelihood of the image including pixels unseen by the deep learning network. ### Claim 11 (depends on 10) The method of claim 10, wherein the unseen metric for the image is based on an aggregate of unseen metrics for each pixel in the image. ### Claim 12 (depends on 11) The method of claim 11, further comprising: generating an alpha metric for the image based a corresponding unseen metric. ### Claim 13 (depends on 11) The method of claim 11, further comprising: generating a threshold metric for the image based on a number of pixels in the image having an unseen score that exceeds a threshold value. ### Claim 14 (depends on 8) The method of claim 8, further comprising: training the prediction model to detect target defects in a target image based on the training data set. ### Claim 15 A computing system, comprising: a processor; and a memory having programming instructions stored thereon, which, when executed by the processor, causes the computing system to perform operations comprising: identifying a set of images for training a prediction model to detect defects in a surface of a specimen, the set of images comprising a first subset of images, wherein each image of the first subset of images is labeled with labels identifying defects on a respective specimen; generating, by a deep learning network of the computing system, a second subset of images for subsequent labelling based on the set of images comprising the first subset of images; prompting an operator to label each image in the second subset of images; generating, by the deep learning network, a third subset of images for labelling based on the set of images comprising the first subset of images and the labeled second subset of images; prompting the operator to label each image in the third subset of images; and aggregating the first subset of images, the second subset of images, and the third subset of images, wherein each image in the first subset of images, the second subset of images, and the third subset of images are labeled to form a training data set for training the prediction model. ### Claim 16 (depends on 15) The computing system of claim 15, wherein generating, by the deep learning network, the second subset of images for subsequent labelling based on the set of images comprising the first subset of images comprises: for each image in the set of images, generating a first probability map corresponding to a first probability of each pixel in the image belongs to a first category; and for each image in the set of images, generating a second probability map corresponding to a second probability of each pixel in the image belongs to a second category. ### Claim 17 (depends on 16) The computing system of claim 16, further comprising: for each image in the set of images, generating an unseen metric based on the first probability map and the second probability map, wherein the unseen metric corresponds to a likelihood of the image including pixels unseen by the deep learning network. ### Claim 18 (depends on 17) The computing system of claim 17, wherein the unseen metric for the image is based on an aggregate of unseen metrics for each pixel in the image. ### Claim 19 (depends on 18) The computing system of claim 18, further comprising: generating an alpha metric for the image based a corresponding unseen metric. ### Claim 20 (depends on 18) The computing system of claim 18, further comprising: generating a threshold metric for the image based on a number of pixels in the image having an unseen score that exceeds a threshold value.