# US 2020/0400425 A1 · Imaging apparatuses, systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0400425 A1
- Document ID: US20200400425A1
- Status: Published application
- Kind: A1
- Application number: 16/948,160
- Filed: 2020-09-04
- Published: 2020-12-24
- Earliest claimed: 2018-03-07
- Inventors: Christian Woehler
- Web page: https://patentgrep.com/patent/US20200400425A1

## CPC

- G01B 11/14: for measuring distance or clearance between spaced objects or spaced apertures (G01B11/26 takes precedence; rangefinders G01C3/00)
- G01B 11/002: {for measuring two or more coordinates}
- G01B 11/026: {by measuring distance between sensor and object (G01B11/0608 takes precedence)}
- G01B 11/22: for measuring depth
- G01N 15/1433: using image recognition
- G01N 21/6428: {Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes" (in vivo A61B5/00; immunoassay G01N33/53)}
- G01N 21/6456: {Spatial resolved fluorescence measurements; Imaging}
- G02B 9/06: two + components
- G02B 13/22: Telecentric objectives or lens systems
- G02B 21/0008: {Microscopes having a simple construction, e.g. portable microscopes}
- G02B 21/361: {Optical details, e.g. image relay to the camera or image sensor (G02B21/364 takes precedence; illumination details G02B21/06 and subgroups)}
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/693: {Acquisition}
- G16B 25/00: ICT specially adapted for hybridisation; ICT specially adapted for gene or protein expression
- G01N 2015/1445: {Three-dimensional imaging, imaging in different image planes, e.g. under different angles or at different depths, e.g. by a relative motion of sample and detector, for instance by tomography}
- G01N 2021/6439: {with indicators, stains, dyes, tags, labels, marks}
- G02B 21/06: Means for illuminating specimens
- H04N 23/55: Optical parts specially adapted for electronic image sensors; Mounting thereof
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image sensor and well structure associated with and extending away from the surface of the image sensor are provided in various apparatuses, methods, and systems for determining the position of a light emitter located in object space. An exemplary method includes (i) providing the image sensor and structure associated therewith, the structure defining a field of view for each pixel within the array of pixels; (ii) determining a light intensity value for photoactivated pixels receiving light from the light emitter; (iii) identifying a first photoactivated pixel having a local maximum of light intensity; (iv) calculating a perpendicular distance between the first photoactivated pixel and the light emitter; and (v) constructing the position of the light emitter based on a position of the first photoactivated pixel in the array of pixels and the perpendicular distance between the first photoactivated pixel and the light emitter.

## Claims

### Claim 1

A computer system for determining a position of a light emitter disposed in object space, comprising:
one or more processors; and
computer readable media having stored thereon computer-executable instructions executable by the one or more processors to cause the system to:
(i) receive—from an image sensor comprising an array of pixels arranged in rows and columns and a structure associated with and extending a height away from a surface of the image sensor, the structure defining a field of view for each pixel within the array of pixels—location information for a plurality of photoactivated pixels within the array of pixels, each of the plurality of photoactivated pixels receiving light from the light emitter disposed in the object space;
(ii) determine a light intensity value for each of the plurality of photoactivated pixels;
(iii) identify a first photoactivated pixel having a local maximum of light intensity, the first photoactivated pixel being closer to the light emitter than other pixels of the plurality of photoactivated pixels receiving less light than the first photoactivated pixel;
(iv) calculate a perpendicular distance between the first photoactivated pixel and the light emitter; and
(v) construct the position of the light emitter based on a position of the first photoactivated pixel in the array of pixels and the perpendicular distance between the first photoactivated pixel and the light emitter.

### Claim 2 (depends on 1)

The system of claim 1, further determining positions of a plurality of light emitters in the object space by causing the computer system to:
(vi) determine a light profile for the light emitter;
(vii) subtract the light profile from light intensity values determined for the plurality of photoactivated pixels; and
(viii) repeat steps (ii)-(vii) for each subsequent light emitter of the plurality of light emitters.

### Claim 3 (depends on 1)

The system of claim 1, wherein calculating the perpendicular distance causes the computer system to:
identify one or more photoactivated pixels receiving light from the light emitter;
identify a light profile for the light emitter based on the light intensity values determined for the first photoactivated pixel and the one or more photoactivated pixels; and
derive the perpendicular distance from the determined light profile for the light emitter.

### Claim 4 (depends on 3)

The system of claim 3, wherein identifying the light profile for the light emitter causes the computer system to:
calculate virtual light intensity values for the first photoactivated pixel and the one or more photoactivated pixels, wherein a virtual light intensity value for a pixel represents the amount of light received by the pixel from a virtual light emitter, the virtual light emitter having the same first photoactivated pixel as the light emitter;
compare the virtual light intensity values with the light intensity values determined for light emitter; and
identify the light profile for the light emitter with the light profile for the virtual light emitter in the case that the virtual light intensity values are fitting to the light intensity values for the light emitter within a tolerance level.

### Claim 5 (depends on 4)

The system of claim 4, wherein the computer system is further caused to calculate the virtual light intensity value for the pixel based on a light receiving area of the pixel receiving light from the virtual light emitter and based on a solid angle of the light receiving area of the pixel with respect to the virtual light emitter.

### Claim 6 (depends on 5)

The system as in claim 5, wherein the computer system is further caused to calculate the virtual light intensity value for the pixel based on a correction factor for the pixel.

### Claim 7 (depends on 4)

The system as in claim 4, wherein the virtual light intensity value for the pixel is determined based on a measured light intensity for the pixel receiving light from a calibration light emitter, the calibration light emitter having a known distance to the image sensor.

### Claim 8 (depends on 5)

The system of claim 5, wherein the virtual light intensity value wi,j of at least one photoactivated pixel pi,j is calculated as:
wi,j=w0·Ωi,j·ai,j,
where:
w0 is a proportionality factor;
Ωi,j is the solid angle of the light receiving area of the at least one photoactivated pixel pi,j with respect to the light emitter; and
αi,j is the light receiving area for the at least one photoactivated pixel pi,j, wherein determining the light receiving area ai,j of the at least one photoactivated pixel pi,j is computed as:
a
i
,
j
=
[
l
-
h
·
(
sgn

(
i
-
i
0
)
·
q
0
,
i
-
l
2
+

i
-
i
0

·
p
)
r
-
h
]
·
[
l
-
h
·
(
sgn

(
j
-
j
0
)
·
q
0
,
j
-
l
2
+

j
-
j
0

·
p
)
r
-
h
]
,
for |i−i0|>0 and |j,−j0|>0 and
for

i
-
i
0

<
[
(
r

l
h
-
sgn

(
i
-
i
0
)
·
q
0
,
i
-
l
2
)
p
]


and



j
-
j
0

<
[
(
r

l
h
-
sgn

(
j
-
j
0
)
·
q
0
,
j
-
l
2
)
p
]
;
a
i
0
,
j
=
[
l
-
h
·
(

q
0
,
i

-
l
2
)
·
step

(

q
0
,
i

-
l
2
)
r
-
h
]
·
[
l
-
h
·
(
sgn

(
j
-
j
0
)
·
q
0
,
j
-
l
2
+

j
-
j
0

·
p
)
r
-
h
]
,
for i=i0 and for |j−j0|>0;
a
i
,
j
0
=
[
l
-
h
·
(
sgn

(
i
-
i
0
)
·
q
0
,
i
-
l
2
+

i
-
i
0

·
p
)
r
-
h
]
·
[
l
-
h
·
(

q
0
,
j

-
l
2
)
·
step
(

q
0
,
j

-
l
2
)
r
-
h
]
,
for j=j0 and for |i−i0|>0; and
a
i
0
,
j
0
=
[
l
-
h
·
(

q
0
,
i

-
l
2
)
·
step

(

q
0
,
i

-
l
2
)
r
-
h
]
·
[
l
-
h
·
(

q
0
,
j

-
l
2
)
·
step

(

q
0
,
j

-
l
2
)
r
-
h
]
,
for i=i0 and j=j0
where:
h is the height of the structure;
l is the row side length of the light sensitive area of the at least one photoactivated pixel, the row side length being equal to the column side length of the light sensitive area of the at least one photoactivated pixel;
p is a pixel pitch of the array of pixels;
r is the nearest distance between the surface of the image sensor and the light emitter;
i0 is the row of the closest photoactivated pixel;
j0 is the column of the closest photoactivated pixel;
q0,i is a distance in the descending row direction between the point on the image sensor surface closest to the light emitter and the center of the closest photoactivated pixel; and
q0,j is a distance in the descending column direction between the point on the image sensor surface closest to the light emitter and the center of the closest photoactivated pixel.

### Claim 9 (depends on 1)

The system of claim 1, wherein constructing the position of the light emitter based on a position of the first photoactivated pixel in the image sensor further causes the computer system to calculate the closest point of the surface of the image sensor to the light emitter to a sub-pixel accuracy by determining differences between light intensity values for a pair of pixels, the pair of pixels being located on a same column or row as the first photoactivated pixel and in different directions from the first photoactivated pixel and having a same distance from the first photoactivated pixel.

### Claim 10 (depends on 1)

The system of claim 1, wherein the structure comprises a plurality of walls positioned on pixel boundaries, forming a regular grid, and the plurality of walls is formed around each pixel.

### Claim 11

A system comprising:
a computer system comprising one or more processors and computer readable media having stored thereon computer-executable instructions executable by the one or more processors to cause the computer system to:
(i) receive—from an image sensor comprising an array of pixels arranged in rows and columns and a structure associated with and extending a height away from a surface of the image sensor, the structure defining a field of view for each pixel within the array of pixels—location information for a plurality of photoactivated pixels, each of the plurality of photoactivated pixels receiving light from a real image of a light emitter located in an object space;
(ii) determine a light intensity value for each of the plurality of photoactivated pixels;
(iii) identify a first photoactivated pixel having a local maximum of light intensity; and
(iv) calculate a perpendicular distance between the first photoactivated pixel having the local maximum and the real image of the light emitter;
(v) construct a position of the real image of the light emitter based on a position of the first photoactivated pixel and the perpendicular distance between the first photoactivated pixel and the real image of the light emitter; and
a lens system placed between the image sensor and the object space,
wherein the real image of the light emitter is being projected by the lens system and the structure comprises a plurality of walls positioned on pixel boundaries, forming a regular grid, and the plurality of walls is formed around each pixel.

### Claim 12 (depends on 11)

The system of claim 11, wherein the computer system is further caused to:
(v) construct a position of the light emitter in the object space based on at least a position of the first photoactivated pixel on the array of pixels, the perpendicular distance between the first photoactivated pixel and the real image of the light emitter, and one or more properties of the lens system.

### Claim 13 (depends on 12)

The system of claim 12, wherein the one or more properties of the lens system comprises a function for relating the position of the real image in image space to the position of the light emitter in the object space.

### Claim 14 (depends on 11)

The system of claim 11, wherein constructing the position of the real image of the light emitter causes the computer system to identify an expected distribution of light intensity that is based on a light distribution of the lens system and that corresponds to a measured distribution of light intensity and to determine the position of the light emitter based on the identified expected distribution.

### Claim 15 (depends on 11)

The system of claim 11, further constructing positions of a plurality of real images of a plurality of light emitters in the object space by causing the computer system to:
(vi) determine a light profile for the light emitter;
(vii) subtract the light profile from light intensity values determined for the plurality of photoactivated pixels; and
(viii) repeat steps (iii)-(vii) for each subsequent light emitter of the plurality of light emitters.

### Claim 16 (depends on 11)

The system of claim 11, wherein calculating the perpendicular distance causes the computer system to:
identify one or more photoactivated pixels receiving light from the light emitter;
identify a light profile for the light emitter based on the light intensity values determined for the first photoactivated pixel and the one or more photoactivated pixels; and
derive the perpendicular distance from the determined light profile for the light emitter,
wherein identifying the light profile for the light emitter causes the computer system to: calculate virtual light intensity values for the first photoactivated pixel and the one or more photoactivated pixels, wherein a virtual light intensity value for a pixel represents the amount of light received by the pixel from a virtual light emitter, the virtual light emitter having the same first photoactivated pixel as the light emitter; compare the virtual light intensity values with the light intensity values determined for light emitter; and identify the light profile for the light emitter with the light profile for the virtual light emitter in the case that the virtual light intensity values are fitting to the light intensity values for the light emitter within a tolerance level.

### Claim 17 (depends on 11)

The system of claim 11, wherein the lens system defines an image space comprising the real image of the light emitter and wherein the real image of the light emitter is located in the field of view of the plurality of pixels and the image sensor records light from the real image of the light emitter, the light emitter located in the object space being in the field of view of the plurality of pixels through the lens system.

### Claim 18

A method for determining a position of a light emitter disposed in object space, comprising:
(i) providing an image sensor comprising an array of pixels arranged in rows and columns and a structure associated with and extending a height away from a surface of the image sensor, the structure defining a field of view for each pixel within the array of pixels;
(ii) determining a light intensity value for each of a plurality of photoactivated pixels, the plurality of photoactivated pixels receiving light from the light emitter disposed in the object space;
(iii) identifying a first photoactivated pixel having a local maximum of light intensity, the first photoactivated pixel being closer to the light emitter than other pixels of the plurality of photoactivated pixels receiving less light than the first photoactivated pixel;
(iv) calculating a perpendicular distance between the first photoactivated pixel and the light emitter; and
(v) constructing the position of the light emitter based on a position of the first photoactivated pixel in the array of pixels and the perpendicular distance between the first photoactivated pixel and the light emitter.

### Claim 19 (depends on 18)

The method as in claim 18, wherein calculating the perpendicular distance comprises:
identifying one or more photoactivated pixels receiving light from the light emitter;
identifying a light profile for the light emitter based on the light intensity values determined for the first photoactivated pixel and the one or more photoactivated pixels; and
deriving the perpendicular distance from the determined light profile for the light emitter.

### Claim 20 (depends on 19)

The method as in claim 19, wherein identifying the light profile for the light emitter comprises:
calculating virtual light intensity values for the first photoactivated pixel and the one or more photoactivated pixels, wherein a virtual light intensity value for a pixel represents the amount of light received by the pixel from a virtual light emitter, the virtual light emitter having the same first photoactivated pixel as the light emitter;
comparing the virtual light intensity values with the light intensity values determined for light emitter; and
identifying the light profile for the light emitter with the light profile for the virtual light emitter in the case that the virtual light intensity values are fitting to the light intensity values for the light emitter within a tolerance level.
