# US 12,217,394 B2 · Method and apparatus for multicolor unmixing through iterative mutual-information minimization in simultaneous channel update structure

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,394 B2
- Document ID: US12217394B2
- Status: Granted
- Kind: B2
- Application number: 17/567,300
- Filed: 2022-01-03
- Published: 2022-07-07
- Granted: 2025-02-04
- Earliest claimed: 2021-01-05
- Assignees: Korea Advanced Institute of Science and Technology
- Applicants: Korea Advanced Institute of Science and Technology
- Inventors: JaeByum Chang; Junyoung Seo; Young-Gyu Yoon; Hyunwoo Kim; Jee Won Kim
- Web page: https://patentgrep.com/patent/US12217394B2

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G01N 21/6428: {Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes" (in vivo A61B5/00; immunoassay G01N33/53)}
- G01N 21/6456: {Spatial resolved fluorescence measurements; Imaging}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/90: Determination of colour characteristics
- G01N 21/6458: {Fluorescence microscopy (fluorescence microscopes per se G02B21/0076 and G02B21/16)}
- G01N 2021/6441: {with two or more labels}
- G06T 2207/10056: Microscopic image
- G06T 2207/10064: Fluorescence image
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)

## Abstract

Methods and apparatuses for multicolor unmixing may involve iterative mutual information minimization in a simultaneous channel update structure. A plurality of images may be obtained with respect to a plurality of fluorophores labeled on different biomolecules, respectively. Images of the fluorophores may be unmixed from the obtained images, respectively, while simultaneously minimizing mutual information shared between the obtained images based on all permutation combinations of the obtained images. Signals having three or more fluorophores whose emission spectra are overlapped in one spectral range can be more accurately unmixed.

## Claims

### Claim 1

A method performed by an electronic device, comprising:
obtaining a plurality of images with respect to a plurality of fluorophores labeled on different biomolecules, respectively; and
unmixing images of the fluorophores from the obtained images, respectively, while simultaneously minimizing mutual information shared between the obtained images based on all permutation combinations (NP2) of the obtained images;
wherein the unmixing of the images of the respective fluorophores comprises:
selecting two images of the obtained images based on one of all the permutation combinations (NP2);
updating an update function wherein updating of the update function comprises processing the selected images as low-resolution images having lower resolution than the selected images as in [Equation i] below; and
u1=IMGi
u2=IMGj
v1=pixelBinning(u1,m)
v2=pixelBinning(u2,m) [Equation i]
wherein the u1 and the u2 indicate the obtained images, the v1 and the v2 indicate the low-resolution images, and the m indicates a factor applied in order to process the obtained images as the low-resolution images, respectively;
calculating the variable from the low-resolution images as in [Equation iii] by using a loss function defined as in [Equation ii] below; and
L
⁡
(
α
)
=
I
⁡
(
v
⁢
⁢
1
;
v
⁢
⁢
2
-
α
×
v
⁢
⁢
1
)
[
Equation
⁢
⁢
ii
]
α
i
,
j
^
=
arg
⁢
⁢
min
α
⁢
⁢
L
⁡
(
α
)
=
arg
⁢
⁢
min
α
⁢
⁢
I
⁡
(
v
⁢
⁢
1
;
v
⁢
⁢
2
-
α
×
v
⁢
⁢
1
)
[
Equation
⁢
⁢
iii
]
wherein the indicates the variable;
updating the update function by applying the variable as in [Equation iv] below;
[
X
1
⁢
(
k
+
1
)
X
2
⁢
(
k
+
1
)
⋮
X
N
⁡
(
k
+
1
)
]
=
[
1
-
γ
⁢
α
^
1
,
2
⁢
(
k
)
…
-
γ
⁢
α
^
1
,
N
⁡
(
k
)
-
γ
⁢
α
^
2
,
1
⁢
(
k
)
1
⋱
⋮
⋮
⋱
⋱
-
γ
⁢
α
^
N
-
1
,
N
⁡
(
k
)
-
γ
⁢
α
^
N
,
1
⁢
(
k
)
…
-
γ
⁢
α
^
N
,
N
-
1
⁢
(
k
)
1
]
⁢
[
X
1
⁢
(
k
)
X
2
⁢
(
k
)
⋮
X
N
⁡
(
k
)
]
[
Equation
⁢
⁢
iv
]
wherein the k indicates an iteration number, the γ indicates an update ratio, and the Xi(k+1) indicates an i-th channel image after a k-th iteration; and
obtaining a plurality of new images updated from the obtained images by using the update function, and Xi(k+1) obtained in a previous step is used as an input image IMGi.

### Claim 2 (depends on 1)

The method of claim 1, wherein the obtaining of the plurality of images comprises obtaining the plurality of images in different detection spectral ranges distinguished from one another, respectively.

### Claim 3 (depends on 2)

The method of claim 2, wherein emission spectra of at least two of the fluorophores are overlapped in each of the detection spectral ranges.

### Claim 4 (depends on 1)

The method of claim 1, wherein the selecting of the two images of the obtained images and updating the update function are iterated based on all the permutation combinations (NP2), and
wherein, when the iteration is completed, the obtaining the plurality of new images updated from the obtained images is performed.

### Claim 5 (depends on 1)

The method of claim 1, wherein the obtaining of the plurality of new images updated from the obtained images is iterated by a predetermined number of times, and
wherein, when the iteration is completed, the obtained images are obtained as the images of the fluorophores, respectively.

### Claim 6 (depends on 1)

The method of claim 1, wherein
the obtaining of the plurality of new images updated from the obtained images is iterated until the mutual information between the obtained images is equal to or smaller than a reference value, and
wherein, when the iteration is completed, the obtained images are obtained as the images of the fluorophores, respectively.

### Claim 7 (depends on 1)

The method of claim 1, wherein the images of the respective fluorophores are obtained as in [Equation v] by imposing positive number restrictions on a solution obtained through the iterative operation;
Xi,k=max(Xi,k,0) [Equation v].

### Claim 8

An electronic device, comprising:
a memory; and
a processor connected to the memory and configured to execute at least one instruction stored in the memory,
wherein the processor is configured to:
obtain a plurality of images with respect to a plurality of fluorophores labeled on different biomolecules, respectively; and
unmix images of the fluorophores from the obtained images, respectively, while simultaneously minimizing mutual information shared between the obtained images based on all permutation combinations (NP2) of the obtained images;
wherein the processor is configured to:
select two images of the obtained images based on one of all the permutation combinations (NP2);
process the selected images as low resolution images having lower resolution than the selected images as in [Equation vi] below; and
u1=IMGi
u2=IMGj
v1=pixelBinning(u1,m)
v2=pixelBinning(u2,m) [Equation vi]
wherein the u1 and the u2 indicate the obtained images, the v1 and the v2 indicate the low resolution images, and the m indicates a factor applied in order to process the obtained images as the low resolution images, respectively;
calculate the variable from the low resolution images as in [Equation viii] by using a loss function defined as in [Equation vii] below; and
L
⁡
(
α
)
=
I
⁡
(
v
⁢
⁢
1
;
v
⁢
⁢
2
-
α
×
v
⁢
⁢
1
)
[
Equation
⁢
⁢
vii
]
=
arg
⁢
⁢
min
α
⁢
⁢
L
⁡
(
α
)
=
arg
⁢
⁢
min
α
⁢
⁢
I
⁡
(
v
⁢
⁢
1
;
v
⁢
⁢
2
-
α
×
v
⁢
⁢
1
)
[
Equation
⁢
⁢
viii
]
wherein the indicates the variable;
updating the update function by applying the variable as in [Equation ix] below;
[
X
1
⁢
(
k
+
1
)
X
2
⁢
(
k
+
1
)
⋮
X
N
⁡
(
k
+
1
)
]
=
[
1
-
γ
⁢
α
^
1
,
2
⁢
(
k
)
…
-
γ
⁢
α
^
1
,
N
⁡
(
k
)
-
γ
⁢
α
^
2
,
1
⁢
(
k
)
1
⋱
⋮
⋮
⋱
⋱
-
γ
⁢
α
^
N
-
1
,
N
⁡
(
k
)
-
γ
⁢
α
^
N
,
1
⁢
(
k
)
…
-
γ
⁢
α
^
N
,
N
-
1
⁢
(
k
)
1
]
⁢
[
X
1
⁢
(
k
)
X
2
⁢
(
k
)
⋮
X
N
⁡
(
k
)
]
[
Equation
⁢
⁢
ix
]
wherein the k indicates an iteration number, the γ indicates an update ratio, and the Xi(k+1) indicates an i-th channel image after a k-th iteration.

### Claim 9 (depends on 8)

The electronic device of claim 8, wherein the processor is configured to obtain the plurality of images in different detection spectral ranges distinguished from one another, respectively.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein emission spectra of at least two of the fluorophores are overlapped in each of the detection spectral ranges.

### Claim 11 (depends on 8)

The electronic device of claim 8, wherein the processor is configured to:
iteratively perform selecting two images of the obtained images and updating the update function based on all the permutation combinations, and
obtain a plurality of new images updated from the obtained images by using the update function when the iteration is completed.

### Claim 12 (depends on 8)

The electronic device of claim 8, wherein the processor is configured to:
iteratively perform obtaining the plurality of new images updated from the obtained images by a predetermined number of times by using the update function, and
obtain the obtained images as the images of the fluorophores, respectively, when completing the iteration.

### Claim 13 (depends on 8)

The electronic device of claim 8, wherein the processor is configured to:
iteratively perform obtaining the plurality of new images updated from the obtained images until the mutual information between the obtained images is equal to or smaller than a reference value by using the update function, and
obtain the obtained images as the images of the fluorophores, respectively, when completing the iteration.

### Claim 14 (depends on 8)

The electronic device of claim 8, wherein the images of the respective fluorophores are obtained as in [Equation x] by imposing positive number restrictions on a solution obtained through the iterative operation;
Xi,k=max(Xi,k,0) [Equation x].

### Claim 15

A non-transitory computer-readable storage medium for storing one or more programs for executing a method, the method comprising:
obtaining a plurality of images with respect to a plurality of fluorophores labeled on different biomolecules, respectively; and
unmixing images of the fluorophores from the obtained images, respectively, while simultaneously minimizing mutual information shared between the obtained images based on all permutation combinations (NP2) of the obtained images;
wherein the unmixing of the images of the respective fluorophores comprises:
selecting two images of the obtained images based on one of all the permutation combinations (NP2);
updating an update function wherein updating of the update function comprises processing the selected images as low-resolution images having lower resolution than the selected images as in [Equation i] below; and
u1=IMGi
u2=IMGj
v1=pixelBinning(u1,m)
v2=pixelBinning(u2,m) [Equation i]
wherein the u1 and the u2 indicate the obtained images, the v1 and the v2 indicate the low-resolution images, and the m indicates a factor applied in order to process the obtained images as the low-resolution images, respectively;
calculating the variable from the low-resolution images as in [Equation iii] by using a loss function defined as in [Equation ii] below; and
L
⁡
(
α
)
=
I
⁡
(
v
⁢
⁢
1
;
v
⁢
⁢
2
-
α
×
v
⁢
⁢
1
)
[
Equation
⁢
⁢
ii
]
α
i
,
j
^
=
arg
⁢
⁢
min
α
⁢
⁢
L
⁡
(
α
)
=
arg
⁢
⁢
min
α
⁢
⁢
I
⁡
(
v
⁢
⁢
1
;
v
⁢
⁢
2
-
α
×
v
⁢
⁢
1
)
[
Equation
⁢
⁢
iii
]
wherein the indicates the variable;
updating the update function by applying the variable as in [Equation iv] below;
[
X
1
⁢
(
k
+
1
)
X
2
⁢
(
k
+
1
)
⋮
X
N
⁡
(
k
+
1
)
]
=
[
1
-
γ
⁢
α
^
1
,
2
⁢
(
k
)
…
-
γ
⁢
α
^
1
,
N
⁡
(
k
)
-
γ
⁢
α
^
2
,
1
⁢
(
k
)
1
⋱
⋮
⋮
⋱
⋱
-
γ
⁢
α
^
N
-
1
,
N
⁡
(
k
)
-
γ
⁢
α
^
N
,
1
⁢
(
k
)
…
-
γ
⁢
α
^
N
,
N
-
1
⁢
(
k
)
1
]
⁢
[
X
1
⁢
(
k
)
X
2
⁢
(
k
)
⋮
X
N
⁡
(
k
)
]
[
Equation
⁢
⁢
iv
]
wherein the k indicates an iteration number, the γ indicates an update ratio, and the Xi(k+1) indicates an i-th channel image after a k-th iteration; and
obtaining a plurality of new images updated from the obtained images by using the update function, and Xi(k+1) obtained in a previous step is used as an input image IMGi.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the images of the respective fluorophores are obtained as in [Equation v] by imposing positive number restrictions on a solution obtained through the iterative operation;
Xi,k=max(Xi,k,0) [Equation v].