# US 2020/0003693 A1 · Sparsity-based super-resolution correlation microscopy

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0003693 A1
- Document ID: US20200003693A1
- Status: Published application
- Kind: A1
- Application number: 16/484,123
- Filed: 2018-01-16
- Published: 2020-01-02
- Earliest claimed: 2017-02-09
- Assignees: Technion Research & Development Foundation Limited
- Applicants: Technion Research & Development Foundation Ltd.
- Inventors: Yonina Eldar; Mordechai Segev; Oren Solomon; Maor Mutzafi
- Web page: https://patentgrep.com/patent/US20200003693A1

## CPC

- G01N 21/6458: {Fluorescence microscopy (fluorescence microscopes per se G02B21/0076 and G02B21/16)}
- G01N 21/64: Fluorescence; Phosphorescence
- G01N 21/6408: {with measurement of decay time, time resolved fluorescence}
- G02B 21/0076: {arrangements using fluorescence or luminescence}
- G02B 27/58: Optics for apodization or superresolution; Optical synthetic aperture systems
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/10: using non-spatial domain filtering
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/73: Deblurring; Sharpening
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10056: Microscopic image
- G06T 2207/10064: Fluorescence image
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An imaging apparatus (20) includes an interface (50) and a processor (30). The interface is configured to receive a sequence of images of an object (38) acquired at respective acquisition times, the object is labeled with fluorescent emitters. The images depict intensity fluctuations of emitters on an acquisition grid, and the intensity fluctuations being uncorrelated among the emitters and during an overall time over which the images in the sequence are acquired. The a processor is configured to calculate a temporal correlation function of the intensity fluctuations over the sequence of images, and to estimate locations of the emitters on a super-resolution grid having a higher resolution than the acquisition grid, by applying to the temporal correlation function a recovery process in which the locations on the super-resolution grid are assumed to be sparse or compressible in a predefined domain. The recovery process includes a sparse-recovery process or a regularized process.

## Claims

### Claim 1

An imaging apparatus, comprising:
an interface, configured to receive a sequence of images of an object acquired at respective acquisition times, wherein the object is labeled with fluorescent emitters, and wherein the images depict intensity fluctuations of emitters on an acquisition grid, the intensity fluctuations being uncorrelated among the emitters and during an overall time over which the images in the sequence are acquired; and
a processor, configured to:
calculate a temporal correlation function of the intensity fluctuations over the sequence of images; and
estimate locations of the emitters on a super-resolution grid having a higher resolution than the acquisition grid, by applying to the temporal correlation function a recovery process in which the locations on the super-resolution grid are assumed to be sparse or compressible in a predefined domain, wherein the recovery process comprises a sparse-recovery process or a regularized process.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein, in applying the recovery process, the processor is configured to distinguish between the estimated locations of emitters that overlap on the acquisition grid.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein the processor is configured to calculate the temporal correlation function by calculating variances of the intensity fluctuations, and to apply the recovery process to the variances to find locations on the super-resolution grid having nonzero variances.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein the processor is configured to transform the received images in the sequence to respective transform-domain vectors, and to calculate the temporal correlation function by calculating a measurement autocorrelation matrix of the transform-domain vectors, for a selected time-lag.

### Claim 5 (depends on 4)

The apparatus according to claim 4, wherein the processor is configured to apply the recovery process by considering one or more nonzero off-diagonal elements of the measurement autocorrelation matrix.

### Claim 6 (depends on 1)

The apparatus according to claim 1, wherein the temporal correlation function and a super-resolution autocorrelation matrix of the intensity fluctuations of the emitters on the super-resolution grid are interrelated by a model matrix representing a resolution degradation in a measurement process of capturing the images in the sequence, and wherein the processor is configured to apply the recovery process by estimating a diagonal vector of a matrix representing the super-resolution autocorrelation matrix.

### Claim 7 (depends on 6)

The apparatus according to claim 6, wherein the recovery process is defined by a optimization problem using an objective function that depends on the temporal correlation function and on the model matrix, and wherein the processor is configured to apply the recovery process by solving the optimization problem under a sparsity or compressibility constraint.

### Claim 8 (depends on 7)

The apparatus according to claim 7, wherein the processor is configured to solve the optimization problem by applying a sequence of iterations for converging to a global minimum of the optimization problem under the sparsity or compressibility constraint, and updating in each iteration a gradient value of the objective function.

### Claim 9 (depends on 7)

The apparatus according to claim 7, wherein a structured matrix derived from the model matrix has a Block Circulant with Circulant Blocks (BCCB) structure, and wherein the processor is configured to update the gradient value by performing a matrix-by-vector multiplication operation with the structured matrix using FFT-based operations.

### Claim 10 (depends on 7)

The apparatus according to claim 7, wherein the optimization problem is formulated under a Total Variation (TV) constraint.

### Claim 11 (depends on 7)

The apparatus according to claim 7, wherein the optimization problem is formulated in a domain in which the solution is sparse or compressible.

### Claim 12 (depends on 7)

The apparatus according to claim 7, wherein the optimization problem comprises a non-convex optimization problem, and wherein the processor is configured to solve the optimization problem using greedy methods.

### Claim 13

A method for imaging, comprising:
receiving a sequence of images of an object acquired at respective acquisition times, wherein the object is labeled with fluorescent emitters, and wherein the images depict intensity fluctuations of emitters on an acquisition grid, the intensity fluctuations being uncorrelated among the emitters and during an overall time over which the images in the sequence are acquired;
calculating a temporal correlation function of the intensity fluctuations over the sequence of images; and
estimating locations of the emitters on a super-resolution grid having a higher resolution than the acquisition grid, by applying to the temporal correlation function a recovery process in which the locations on the super-resolution grid are assumed to be sparse or compressible in a predefined domain, wherein the recovery process comprises a sparse-recovery process or a regularized process.

### Claim 14 (depends on 13)

The method according to claim 13, wherein, in applying the recovery process, distinguishing between the estimated locations of emitters that overlap on the acquisition grid.

### Claim 15 (depends on 13)

The method according to claim 13, wherein calculating the temporal correlation function comprises calculating variances of the intensity fluctuations, and wherein applying the recovery process comprises applying the recovery process to the variances to find locations on the super-resolution grid having nonzero variances.

### Claim 16 (depends on 13)

The method according to claim 13, wherein calculating the temporal correlation function comprises transforming the received images in the sequence to respective transform-domain vectors, and calculating the temporal correlation function by calculating a measurement autocorrelation matrix of the transform-domain vectors, for a selected time-lag.

### Claim 17 (depends on 16)

The method according to claim 16, wherein applying the recovery process comprises considering one or more nonzero off-diagonal elements of the measurement autocorrelation matrix.

### Claim 18 (depends on 13)

The method according to claim 13, wherein the temporal correlation function and a super-resolution autocorrelation matrix of the intensity fluctuations of the emitters on the super-resolution grid are interrelated by a model matrix representing a resolution degradation of a measurement process of capturing the images in the sequence, and wherein applying the recovery process comprises estimating a diagonal vector of a matrix representing the super-resolution autocorrelation matrix.

### Claim 19 (depends on 18)

The method according to claim 18, wherein the recovery process is defined by an optimization problem using an objective function that depends on the temporal correlation function and on the model matrix, and wherein applying the recovery process comprises solving the optimization problem under a sparsity or compressibility constraint.

### Claim 20 (depends on 19)

The method according to claim 19, wherein solving the optimization problem comprises applying a sequence of iterations for converging to a global minimum of the optimization problem under the sparsity or compressibility constraint, and updating in each iteration a gradient value of the objective function.

### Claim 21 (depends on 19)

The method according to claim 19, wherein a structured matrix derived from the model matrix has a Block Circulant with Circulant Blocks (BCCB) structure, and wherein solving the optimization problem comprises updating the gradient value by performing a matrix-by-vector multiplication operation with the structured matrix using FFT-based operations.

### Claim 22 (depends on 19)

The method according to claim 19, wherein the optimization problem is formulated under a Total Variation (TV) constraint.

### Claim 23 (depends on 19)

The method according to claim 19, wherein the optimization problem is formulated in a domain in which the solution is sparse or compressible.

### Claim 24 (depends on 19)

The method according to claim 19, wherein the optimization problem comprises a non-convex optimization problem, and wherein solving the optimization problem comprises solving the optimization problem using greedy methods.
