# US 2022/0277441 A1 · Image processing apparatus and method for use in an autofocus system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277441 A1
- Document ID: US20220277441A1
- Status: Published application
- Kind: A1
- Application number: 17/637,480
- Filed: 2020-08-31
- Published: 2022-09-01
- Earliest claimed: 2019-08-30
- Assignees: Leica Microsystems CMS GmbH
- Applicants: Leica Microsystems Cms GmbH
- Inventors: Falk SCHLAUDRAFF; Markus SCHECHTER; Frank SIECKMANN; Oliver SCHLICKER
- Web page: https://patentgrep.com/patent/US20220277441A1

## CPC

- G06T 5/73: Deblurring; Sharpening
- G02B 7/38: measured at different points on the optical axis {, e.g. focussing on two or more planes and comparing image data}
- G02B 21/12: affording bright-field illumination (G02B21/14 takes precedence)
- G02B 21/244: {using image analysis techniques}
- G03B 13/36: Autofocus systems
- G06F 17/13: Differential equations (using digital differential analysers G06F7/64)
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06T 5/10: using non-spatial domain filtering
- G06T 5/20: using local operators
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/30: Noise filtering
- H04L 25/025: {using least-mean-square [LMS] method}
- H04N 25/615: involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
- G01S 7/417: {involving the use of neural networks}
- G01S 13/89: for mapping or imaging
- G01S 15/89: for mapping or imaging
- G01V 1/36: Effecting static or dynamic corrections on records, e.g. correcting spread; Correlating seismic signals; Eliminating effects of unwanted energy
- G02B 3/14: of variable focal length
- G02B 21/086: {for transillumination only}
- G02B 21/241: {Devices for focusing (focusing in general G02B7/28)}
- G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)}
- G06T 2207/10056: Microscopic image
- G06T 2207/10068: Endoscopic image
- G06T 2207/10072: Tomographic images
- G06T 2207/10132: Ultrasound image
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01V: GEOPHYSICS; GRAVITATIONAL MEASUREMENTS; DETECTING MASSES OR OBJECTS; TAGS (means for indicating the location of accidentally buried, e.g. snow-buried, persons A63B29/02)

## Abstract

An image processing apparatus for determining a focused output image in a passive autofocus system is configured to retrieve a set of input images and compute a baseline estimate for at least one input image. The baseline estimate represents image structures in the input image. The image structures have a length scale larger than a predetermined image feature length scale. The image processing apparatus is further configured to compute a set of output images, wherein each output image of the set of output images is computed based on one of a different input image of the set of input images and the at least one baseline estimate for the different input image and the at least one baseline estimate for a respective different input image. The image processing apparatus is further configured to determine one output image of the set of output images as the focused output image.

## Claims

### Claim 1

An image processing apparatus for determining a focused output image a passive autofocus system, the image processing apparatus being configured to:
retrieve a set of input images;
compute at least one baseline estimate for at least one input image of the set of input images, the at least one baseline estimate representing image structures in the at least one input image, the image structures having a length scale larger than a predetermined image feature length scale;
compute a set of output images;
compute each output image of the set of output images based on one of a different input image of the set of input images and the at least one baseline estimate for the different input image and the at least one baseline estimate for a respective different input image; and
determine one output image of the set of output images as the focused output image.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein the image processing apparatus is configured to compute each output image of the set of output images based on a different input image and the at least one baseline estimate for the different input image, and is further configured to:
remove the at least one baseline estimate from the respective input image to obtain an output image of the set of output images.

### Claim 3 (depends on 1)

The image processing apparatus according to claim 1, wherein the image processing apparatus is further configured to:
compute for an input image of the set of input images a first baseline estimate based on a first predetermined image feature length scale and a second baseline estimate based on a second image feature length scale, the second image feature length scale being different from the first image feature length scale, and
compute an output image based on the first and the second baseline estimate.

### Claim 4 (depends on 3)

The image processing apparatus according to claim 3, wherein the image processing apparatus is further configured to:
remove one of the baseline estimates based on a larger image feature length scale from one of the baselines estimate based on a smaller image feature length scale.

### Claim 5 (depends on 1)

The image processing apparatus according to claim 1, wherein the image processing apparatus is further configured to:
obtain a baseline estimate as an output image of the set of output images.

### Claim 6 (depends on 1)

The image processing apparatus according to claim 1, wherein the image processing apparatus is further configured to:
compute the baseline estimate using a least-square minimization criterion for the baseline estimate, the least-square minimization criterion comprising a scalar combination of the image feature length scale and a derivative of the baseline estimate.

### Claim 7 (depends on 6)

The image processing apparatus according to claim 6, wherein the least-square minimization criterion comprises a penalty term, the penalty term containing the image feature length scale.

### Claim 8 (depends on 1)

The image processing apparatus according to claim 1, wherein the image processing apparatus is further configured to use a focus function to determine the one output image of the set of output images as the focused image.

### Claim 9 (depends on 8)

The image processing apparatus according to claim 8, wherein the focus function comprises at least one focus function of the following list:
computation of an amount of entropy contained in the input image and/or the output image;
computation of a contrast in at least a part of the input image and/or the output image;
computation of an intensity and/or an intensity distribution in at least part of the input image and/or the output image;
computation of a phase correlation; and
computation of a correlation with a predetermined pattern.

### Claim 10 (depends on 1)

The image processing apparatus according to claim 1, wherein the image processing apparatus is further configured to pre-process at least one input image of the set of input images using a top-hat transform.

### Claim 11 (depends on 1)

The image processing apparatus according claim 1, wherein the image processing apparatus is further configured to pre-process at least one input image of the set of input images to improve image quality and/or for filtering.

### Claim 12 (depends on 1)

An autofocus system comprising the image processing apparatus according to claim 1, and an imaging system comprising an image sensor and an autofocus objective.

### Claim 13 (depends on 1)

An observation device comprising the image processing apparatus according to claim 1, and a display, the display being configured to display the focused output image.

### Claim 14 (depends on 13)

The observation device according to claim 13, the observation device being one of a microscope and an endoscope.

### Claim 15

A computer-implemented image processing method for determining a focused output image in a passive autofocus system, the method comprising:
retrieving a set of input images;
computing at least one baseline estimate for at least one input image of the set of input images, the at least one baseline estimate representing image structures having a length scale larger than a predetermined image feature length scale;
computing a set of output images;
the step computing the set of output images comprising the step of:
computing each output image of the set of output images based on one of a different input image of the set of input images and the at least one baseline estimate for the different input image and the at least one baseline estimate for a respective different input image; and
selecting one output image of the set of output images as the focused output image.

### Claim 16 (depends on 15)

The computer-implemented image processing method according to claim 15, wherein the step of retrieving the set of input images comprises the step of automatically capturing at least some input images of the set of input images at different focus distances and/or different relative positions of a field of view and an object.

### Claim 17 (depends on 15)

A computer-implemented autofocusing method, comprising the image processing method of claim 15 and a step of displaying the selected output image.

### Claim 18

(canceled)

### Claim 19 (depends on 15)

A tangible, non-transitory computer readable medium storing a computer program causing a computer to execute the image processing method according to claim 15 when the computer program is executed by the computer.

### Claim 20 (depends on 15)

A machine learning device configured for use in an autofocus system, trained by sets of input images and selected output images, wherein the selected output images are created from the set of input images by the method according to claim 15.

### Claim 21 (depends on 15)

A focused output image comprising the selected output image that results from the method according to claim 15.
