# US 12,002,185 B2 · Analyzing complex single molecule emission patterns with deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,185 B2
- Document ID: US12002185B2
- Status: Granted
- Kind: B2
- Application number: 17/309,027
- Filed: 2019-06-10
- Published: 2022-01-20
- Granted: 2024-06-04
- Earliest claimed: 2018-10-17
- Assignees: Purdue Research Foundation
- Applicants: Purdue Research Foundation
- Inventors: Peiyi Zhang; Fang Huang; Sheng Liu
- Web page: https://patentgrep.com/patent/US12002185B2

## CPC

- G06T 3/4046: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G01J 3/2823: {Imaging spectrometer}
- G01J 3/443: Emission spectrometry
- G01N 21/6428: {Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes" (in vivo A61B5/00; immunoassay G01N33/53)}
- G01N 21/6458: {Fluorescence microscopy (fluorescence microscopes per se G02B21/0076 and G02B21/16)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G01N 2201/1296: using neural networks
- G06F 2218/02: Preprocessing
- G06F 2218/22: Source localisation; Inverse modelling
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A fluorescent single molecule emitter simultaneously transmits its identity, location, and cellular context through its emission patterns. A deep neural network (DNN) performs multiplexed single-molecule analysis to enable retrieving such information with high accuracy. The DNN can extract three-dimensional molecule location, orientation, and wavefront distortion with precision approaching the theoretical limit of information content of the image which will allow multiplexed measurements through the emission patterns of a single molecule.

## Claims

### Claim 1

A computer-implemented method for training a neural network for use in extracting physical information from point spread function (PSF) emission patterns of illuminated single molecules, the emission patterns corresponding to ground truth parameters θtn indicative of the physical information of the molecule, the method comprising:
(a) acquiring a dataset of a plurality of N×N pixel images An of the PSF emission patterns and providing the dataset to a processor;
(b) forward propagating the dataset through a neural network, with the processor, comprising a plurality of convolutional and residual layers l and a plurality of training parameters wl in the layers l, to generate an output vector of a number of physical parameters {circumflex over (θ)}tn in a kth iteration through the neural network;
(c) comparing, with the processor, the output physical parameters {circumflex over (θ)}tn to the ground truth parameters θtn by calculating a Cramér-Rao lower bound (CRLB)-weighted mean squared error (MSE) based on the equation
E
θ
^
=
1
NT
⁢
∑
n
=
1
N
⁢
∑
t
=
1
T
⁢
(
θ
^
tn
-
θ
tn
)
2
Measure
θ
tn
,
where N is the number of images An, T is the size of the output vector for each image, and CRLBeθtn is the value of the Cramér-Rao lower bound (CRLB) for parameter θt of image An;
(d) updating, with the processor, the parameters wl of the layers l of the neural network based on the derivative of the CRLB-weighted MSE according to the equation
w
k
+
1
l
=
w
k
l
-
η
M
⁢
∑
n
=
1
M
⁢
∂
E
θ
^
⁢
n
,
k
∂
w
n
,
k
l
,
where l=0,1,2, . . . L denotes the layers in the neural network, η is the learning rate, k is the iteration number, and M is the number of images An;
(e) repeating steps (b)-(d) until the error E{circumflex over (θ)} reaches a predetermined limit, after which the parameters wl of the layers l of the neural network are assigned, with the processor, the updated values of the last iteration for subsequently using the neural network, with the processor, to identify unknown physical parameters from the PSF of new image of a single molecule.

### Claim 2 (depends on 1)

The method of claim 1, wherein the physical parameters include one or more of axial (z) position, lateral (x, y) position, wavefront distortion and dipole orientation of the single molecule.

### Claim 3 (depends on 2)

The method of claim 2, wherein the output vector includes at least two physical parameters.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of layers of the neural network includes a fully connected layer between the output and the last one of the convolutional and residual layers.

### Claim 5 (depends on 4)

The method of claim 4, wherein the plurality of layers of the neural network includes a Hard Tanh layer between the fully connected layer and the output.

### Claim 6 (depends on 1)

The method of claim 1, wherein each residual layer includes a PreLU (Parametric Rectified Linear Unit) activation function at the output of the residual layer.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
dividing the acquired dataset into a training dataset and a validation dataset;
forward propagating both datasets through said neural network;
for both datasets, calculating the mean squared error weighted by the information content;
comparing the error for the validation dataset to said predetermined limit.

### Claim 8 (depends on 1)

The method of claim 1, further comprising the step of shuffling the order of the images within the dataset prior to forward propagating the dataset through the neural network for each of the successive iterations.

### Claim 9

A computer-implemented method for identifying unknown physical parameters from the point spread function (PSF) of an image of a single molecule, the physical parameters including one or more of axial (z) position, lateral (x, y) position, wavefront distortion and dipole orientation of the single molecule, the method comprising:
(a) training a neural network for use in extracting physical information from PSF emission patterns of illuminated single molecules, the emission patterns corresponding to ground truth parameters θtn indicative of the known physical information of the molecule, by;
(i) acquiring a dataset of a plurality of N×N pixel images An of the PSF emission patterns corresponding to ground truth parameters θtn and providing the dataset to a processor;
(ii) forward propagating the dataset through a neural network, with the processor, comprising a plurality of convolutional and residual layers l and a plurality of training parameters wl in the layers l, to generate an output vector of a number of physical parameters {circumflex over (θ)}tn in a kth iteration through the neural network;
(iii) comparing, with the processor, the output physical parameters {circumflex over (θ)}tn to the ground truth parameters θtn and calculating a Cramér-Rao lower bound (CRLB)-weighted mean squared error (MSE) based on the equation
E
θ
^
=
1
NT
⁢
∑
n
=
1
N
⁢
∑
t
=
1
T
⁢
(
θ
^
tn
-
θ
tn
)
2
CRLB
θ
tn
,
where N is the number of images An, T is the size of the output vector for each image, and CRLBeθtn is the value of the Cramér-Rao lower bound (CRLB) for parameter θt of image An;
(iv) updating, with the processor, the parameters wl of the layers l of the neural network based on the derivative of the CRLB-weighted MSE according to the equation
w
k
+
1
l
=
w
k
l
-
η
M
⁢
∑
n
=
1
M
⁢
∂
E
θ
^
⁢
n
,
k
∂
w
n
,
k
l
,
where l=0,1,2, . . . L denotes the layers in the neural network, η is the learning rate, k is the iteration number, and M is the number of images An;
(v) repeating steps (ii)-(iv) until the error E{circumflex over (θ)} reaches a predetermined limit, after which the parameters wl of the layers l of the neural network are assigned, with the processor, the updated values of the last iteration; and
(b) acquiring a new dataset of an N×N pixel images of the PSF emission patterns corresponding to the unknown physical parameters and providing the dataset to the processor; and
(c) forward propagating the new dataset through the trained neural network to generate an output vector with values corresponding to the one or more physical parameters.