# US 2022/0044074 A1 · Efficient Training and Accuracy Improvement of Imaging Based Assay

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044074 A1
- Document ID: US20220044074A1
- Status: Published application
- Kind: A1
- Application number: 17/431,345
- Filed: 2020-11-25
- Published: 2022-02-10
- Earliest claimed: 2019-11-25
- Assignees: Essenlix Corporation
- Applicants: Essenlix Corporation
- Inventors: Xing Li; Wu CHOU; Stephen Y. CHOU; Wei DING
- Web page: https://patentgrep.com/patent/US20220044074A1

## CPC

- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/094: Adversarial learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/11: Region-based segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30204: Marker
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure relates to devices, apparatus and methods of improving the accuracy of image-based assay, that uses imaging system having uncertainties or deviations (imperfection) compared with an ideal imaging system. One aspect of the present invention is to add the monitoring marks on the sample holder, with at least one of their geometric and/optical properties of the monitoring marks under predetermined and known, and taking images of the sample with the monitoring marks, and train a machine learning model using the images with the monitoring mark.

## Claims

### Claim 1

A method of training a machine learning model for an image based assay, wherein the assay is, during a test, imaged by a low-quality imaging system, comprising:
having a first sample forming a thin layer on an imaging area of a first sample holder, wherein the first sample holder is a marked sample holder comprising one or more monitoring marks on the imaging area;
having a second sample forming a thin layer on an imaging area of a second sample holder, wherein the second sample holder is a marked sample holder that comprising an identical one or more monitoring marks on the imaging area of the second sample holder to the one or more monitoring marks on the first sample holder;
imaging, using a low-quality imaging system, a first image of the sample on the imaging area of the first sample holder;
imaging; using a high-quality imaging system, a second image of the sample on the imaging area of the second sample holder;
correcting an imperfection in the first image using the monitoring marks, generating a first corrected image;
correcting an imperfection in the second image using the monitoring marks, if the second image has an imperfection, generating a second corrected image; and
training a machine learning model using the first corrected image, the second corrected image and the monitoring marks, generating a trained model, wherein a geometric property and optionally an optical property of the one or more monitoring marks imaged under an ideal imaging system is predetermined and known;
wherein a low quality imaging system comprise more imperfection than a high quality imaging system.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
having a third sample forming a thin layer on an imaging area of a third sample holder, wherein the third sample holder is a marked sample holder that comprising one or more monitoring marks on the imaging area of the third sample holder identical to the one or more monitoring marks on the first sample holder;
imaging, using a low-quality imaging system, a third image of the samples on the imaging area of the third sample holder;
correcting, using the monitoring marks, an imperfection in the third image, generating a corrected third image; and
analyzing the transformed corrected third image using the trained machine learning model trained and generating an assay result.

### Claim 3 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a cycle generative adversarial network (CycleGAN).

### Claim 4 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a cycle generative adversarial network (CycleGAN) comprising a forward generative adversarial network (forward GAN) and a backward GAN, wherein the forward GAN comprises a first generator and a first discriminator, and the backward GAN comprises a second generator and a second discriminator, and wherein training the machine learning model using each of transformed regions in the first image and each of transformed regions in the second image comprises training the CycleGAN using each of transformed regions in the first image and each of transformed regions in the second image registered at four structural elements at four corners of the corresponding regions.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first sample and the second sample are the same sample, and the first sample holder and the second sample holder are the same.

### Claim 6

A method to train a machine learning model for image-based assays, the method comprising:
receiving a first image, captured by a first optical sensor, of a sample holder containing a sample, wherein the sample holder is fabricated with a standard of patterned structural elements at predetermined positions;
identifying a first region in the first image based on locations of one or more structural elements of the patterned structural elements in the first image;
determining a spatial transform associated with the first region based on a mapping between the locations of the one or more structural elements in the first image and predetermined positions of one or more structural elements in the sample holder;
applying the spatial transform to the first region in the first image to calculate a transformed first region; and
training the machine learning model using the transformed first image.

### Claim 7 (depends on 6)

The method of claim 6, wherein the sample holder comprises a first plate, a second plate, and the patterned structural elements, and wherein the patterned structural elements comprise pillars embedded at the predetermined positions on at least one of the first plate or the second plate.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
detecting the locations of the patterned structural elements in the first image;
partitioning the first image into regions comprising the first region, wherein each of the regions is defined by four structural elements at four corners of the corresponding region;
determining a corresponding spatial transform associated with each of the regions in the first image based on a mapping between the locations of the four structural elements at the four corners of the corresponding region and the four predetermined positions of the four structural elements in the sample holder;
applying the corresponding spatial transform to each of the regions in the first image to calculate a corresponding transformed region in the first image; and
training the machine learning model using each of transformed regions in the first image, wherein the trained machine learning model is used to transform assay images from a low resolution to a high resolution.

### Claim 9 (depends on 8)

The method of claim 8, wherein the predetermined positions of the patterned structural elements are distributed periodically with at least one periodicity value, and wherein detecting the locations of the patterned structural elements in the first image comprises:
detecting, using a second machine learning model, the locations of the patterned structural elements in the first image; and
correcting, based on the at least one periodicity value, an error in the detected locations of the patterned structural elements in the first image.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
receiving a second image of the sample holder captured by a second optical sensor, wherein the first image is captured at a first quality level and the second image is captured at a second quality level which is higher than the first quality level;
partitioning the second image into regions, wherein each of the regions in the second image is defined by four structural elements at four corners of the corresponding region in the second image and is matched to a corresponding region in the first image;
determining a second spatial transform associated with a region in the second image based on a mapping between the locations of the four structural elements at the four corners of the corresponding region in the second image and the four predetermined positions of the four structural elements in the sample holder;
applying the second spatial transform to each of the regions in the second image to calculate a corresponding transformed region in the second image; and
training the machine learning model from transforming first quality level images to second quality level images using each of transformed regions in the first image and each of transformed regions in the second image.

### Claim 11 (depends on 10)

The method of claim 10, wherein the machine learning model comprises a cycle generative adversarial network (CycleGAN) comprising a forward generative adversarial network (forward GAN) and a backward GAN, wherein the forward GAN comprises a first generator and a first discriminator, and the backward GAN comprises a second generator and a second discriminator, and wherein training the machine learning model using each of transformed regions in the first image and each of transformed regions in the second image comprises training the CycleGAN using each of transformed regions in the first image and each of transformed regions in the second image registered at four structural elements at four corners of the corresponding regions.

### Claim 12 (depends on 11)

The method of claim 11, wherein training the machine learning model using each of transformed regions in the first image and each of transformed regions in the second image comprises:
training the first generator and the first discriminator by providing each of transformed regions in the first image to the forward GAN;
training the second generator and the second discriminator by providing each of transformed regions in the second image to the backward GAN and
optimizing the forward and backward GAN training under a cycle consistency constraint.

### Claim 13

A method for converting an assay image using a machine learning model, the method comprising:
receiving a first image, captured by a first optical sensor, of a sample holder containing a sample, wherein the sample holder is fabricated with a standard of patterned structural elements at predetermined positions;
identifying a first region in the first image based on locations of one or more structural elements of the patterned structural elements in the first image;
determining a spatial transform associated with the first region based on a mapping between the locations of the one or more structural elements in the first image and predetermined positions of one or more structural elements in the sample holder;
applying the spatial transform to the first region in the first image to calculate a transformed first region; and
applying the machine learning model to the transformed first region in the first image to generate a second region.

### Claim 14 (depends on 13)

The method of claim 13, wherein the machine learning model is trained by performing:
receiving a first image, captured by a first optical sensor, of a sample holder containing a sample, wherein the sample holder is fabricated with a standard of patterned structural elements at predetermined positions;
identifying a first region in the first image based on locations of one or more structural elements of the patterned structural elements in the first image;
determining a spatial transform associated with the first region based on a mapping between the locations of the one or more structural elements in the first image and predetermined positions of one or more structural elements in the sample holder;
applying the spatial transform to the first region in the first image to calculate a transformed first region; and
training the machine learning model using the transformed first image.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
partitioning the first image into a plurality of regions based the locations of the one or more structural elements of the patterned structural elements in the first image, wherein the plurality of regions comprises the first region;
determining a respective spatial transform associated with each of the plurality of regions;
applying the corresponding spatial transform to each of the plurality of regions in the first image to calculate transformed regions;
applying the machine learning model to each of the transformed regions in the first image to generate transformed regions of a second quality level; and
combining the transformed regions to form a second image.

### Claim 16

An image-based assay system, comprising:
a database system to store images; and
a processing device, communicatively coupled to the database system, to:
receive a first image, captured by a first optical sensor, of a sample holder containing a sample, wherein the sample holder is fabricated with a standard of patterned structural elements at predetermined positions;
identify a first region in the first image based on locations of one or more structural elements of the patterned structural elements in the first image;
determine a spatial transform associated with the first region based on a mapping between the locations of the one or more structural elements in the first image and predetermined positions of one or more structural elements in the sample holder;
apply the spatial transform to the first region in the first image to calculate a transformed first region; and
train the machine learning model using the transformed first image.

### Claim 17 (depends on 16)

The system of claim 16, wherein the sample holder comprises a first plate, a second plate, and the patterned structural elements, and wherein the patterned structural elements comprise pillars embedded at the predetermined positions on at least one of the first plate or the second plate.

### Claim 18 (depends on 16)

The system of claim 16, wherein the processing device is further to:
detect the locations of the patterned structural elements in the first image;
partition the first image into regions comprising the first region, wherein each of the regions is defined by four structural elements at four corners of the corresponding region;
determine a corresponding spatial transform associated with each of the regions in the first image based on a mapping between the locations of the four structural elements at the four corners of the corresponding region and the four predetermined positions of the four structural elements in the sample holder;
apply the corresponding spatial transform to each of the regions in the first image to calculate a corresponding transformed region in the first image; and
train the machine learning model using each of transformed regions in the first image, wherein the trained machine learning model is used to transform assay images from a low resolution to a high resolution.

### Claim 19 (depends on 18)

The system of claim 18, wherein the predetermined positions of the patterned structural elements are distributed periodically with at least one periodicity value, and wherein to detect the locations of the patterned structural elements in the first image, the processing device is further to:
detect, using a second machine learning model, the locations of the patterned structural elements in the first image; and
correct, based on the at least one periodicity value, an error in the detected locations of the patterned structural elements in the first image.

### Claim 20 (depends on 18)

The system of claim 18, wherein the processing device is further to:
receive a second image of the sample holder captured by a second optical sensor, wherein the first image is captured at a first quality level and the second image is captured at a second quality level which is higher than the first quality level;
partition the second image into regions, wherein each of the regions in the second image is defined by four structural elements at four corners of the corresponding region in the second image and is matched to a corresponding region in the first image;
determine a second spatial transform associated with a region in the second image based on a mapping between the locations of the four structural elements at the four corners of the corresponding region in the second image and the four predetermined positions of the four structural elements in the sample holder;
apply the second spatial transform to each of the regions in the second image to calculate a corresponding transformed region in the second image; and
train the machine learning model from transforming first quality level images to second quality level images using each of transformed regions in the first image and each of transformed regions in the second image.

### Claim 21 (depends on 20)

The system of claim 20, wherein the machine learning model comprises a cycle generative adversarial network (CycleGAN) comprising a forward generative adversarial network (forward GAN) and a backward GAN, wherein the forward GAN comprises a first generator and a first discriminator, and the backward GAN comprises a second generator and a second discriminator, and wherein training the machine learning model using each of transformed regions in the first image and each of transformed regions in the second image comprises training the CycleGAN using each of transformed regions in the first image and each of transformed regions in the second image registered at four structural elements at four corners of the corresponding regions.

### Claim 22 (depends on 21)

The system of claim 21, wherein to train the machine learning model using each of transformed regions in the first image and each of transformed regions in the second image, the processing device is further to:
train the first generator and the first discriminator by providing each of transformed regions in the first image to the forward GAN; and
train the second generator and the second discriminator by providing each of transformed regions in the second image to the backward GAN.

### Claim 23

An image-based assay system for converting an assay image using a machine learning model, comprising:
a database system to store images; and
a processing device, communicatively coupled to the database system, to:
receive a first image, captured by a first optical sensor, of a sample holder containing a sample, wherein the sample holder is fabricated with a standard of patterned structural elements at predetermined positions;
identify a first region in the first image based on locations of one or more structural elements of the patterned structural elements in the first image;
determine a spatial transform associated with the first region based on a mapping between the locations of the one or more structural elements in the first image and predetermined positions of one or more structural elements in the sample holder;
apply the spatial transform to the first region in the first image to calculate a transformed first region; and
apply the machine learning model to the transformed first region in the first image to generate a second region.

### Claim 24 (depends on 23)

The system of claim 23, wherein the machine learning model is trained by performing:
receiving a first image, captured by a first optical sensor, of a sample holder containing a sample, wherein the sample holder is fabricated with a standard of patterned structural elements at predetermined positions;
identifying a first region in the first image based on locations of one or more structural elements of the patterned structural elements in the first image;
determining a spatial transform associated with the first region based on a mapping between the locations of the one or more structural elements in the first image and predetermined positions of one or more structural elements in the sample holder;
applying the spatial transform to the first region in the first image to calculate a transformed first region; and
training the machine learning model using the transformed first image.

### Claim 25 (depends on 23)

The system of claim 23, wherein the processing device is further to:
partition the first image into a plurality of regions based the locations of the one or more structural elements of the patterned structural elements in the first image, wherein the plurality of regions comprises the first region;
determine a respective spatial transform associated with each of the plurality of regions;
apply the corresponding spatial transform to each of the plurality of regions in the first image to calculate transformed regions;
apply the machine learning model to each of the transformed regions in the first image to generate transformed regions of a second quality level; and
combine the transformed regions to form a second image.

### Claim 26 (depends on 1)

The method of claim 1, wherein the sample holder has one plate and the sample is contacting one surface of the plate.

### Claim 27 (depends on 1)

The method of claim 1, wherein the sample holder has two plates, and the sample is between the two plates.

### Claim 28 (depends on 1)

The method of claim 1, wherein the sample holder has two plates that are movable relative to each other, and the sample is between the two plates.

### Claim 29 (depends on 1)

The method of claim 1, wherein:
the sample holder has two plates that is movable to each other, and the sample is between the two plates;
a plurality of spacers are attached to at least one interior opposing surface of at least one of the plates, or both, and the plurality of spacers are situated between the opposable plates; and
the sample thickness is regulated by the spacers.

### Claim 30 (depends on 1)

The method of claim 1, wherein there is at least one spacer inside the sample.

### Claim 31 (depends on 1)

The method of claim 1, wherein the spacers are the monitoring markers.

### Claim 32 (depends on 1)

The method of claim 1, wherein the two plates of the device are initially on top of each other and need to be separated to get into an open configuration for sample deposition.

### Claim 33 (depends on 1)

The method of claim 1, wherein the two plates of the device are in the closed configuration before the sample deposition and the sample gets into the device from a gap between the two plates.

### Claim 34 (depends on 1)

The method of claim 1, wherein the thickness of the sample layer is 0.1 um, 0.5 um, 1 um, 2 um, 3 um, 5 um, 10 um, 50 um, 100 um, 200 um, 500 um, 1000 um, 5000 um or in a range between any of the two values.

### Claim 35 (depends on 1)

The method of claim 1, wherein the preferred thickness of the sample layer is 1 um, 2 um, 5 um, 10 um, 30 um, 50 um, 100 um, 200 um, or in a range between any of the two values.

### Claim 36 (depends on 1)

The method of claim 1, wherein the spacing between two monitoring markers is 1 um, 2 um, 3 um, 5 um, 10 um, 50 um, 100 um, 200 um, 500 um, 1000 um, 5000 um or in a range between any of the two values.

### Claim 37 (depends on 1)

The method of claim 1, wherein the preferred spacing between two monitoring markers is 10 um, 50 um, 100 um, 200 um, or in a range between any of the two values.
