# US 2023/0055377 A1 · Automated training of a machine-learned algorithm on the basis of the monitoring of a microscopy measurement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0055377 A1
- Document ID: US20230055377A1
- Status: Published application
- Kind: A1
- Application number: 17/889,550
- Filed: 2022-08-17
- Published: 2023-02-23
- Earliest claimed: 2021-08-20
- Assignees: Carl Zeiss Microscopy GmbH
- Applicants: Carl Zeiss Microscopy GmbH
- Inventors: Manuel AMTHOR; Daniel HAASE
- Web page: https://patentgrep.com/patent/US20230055377A1

## CPC

- G06T 7/11: Region-based segmentation
- G06V 20/698: {Matching; Classification}
- G06T 7/60: Analysis of geometric attributes
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/693: {Acquisition}
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented method comprises the following steps. In one step an image is acquired which is captured in the context of a microscopy measurement and images a sample to be examined. In one step the microscopy measurement is monitored in an automated manner. On the basis of the automated monitoring of the microscopy measurement, one or more labels are created, wherein said one or more labels comprise semantic context information of the microscopy measurement. On the basis of the image as input and said one or more labels as ground truth, a machine-learned algorithm is trained which provides semantic context information on the basis of images captured in the context of microscopy measurements. In a further step a further image is acquired, which is captured in the context of the microscopy measurement or a further microscopy measurement by the microscope and images the sample or a further sample. In a further step the trained machine-learned algorithm is applied to the further image in order to predict further semantic context information for the further image.

## Claims

### Claim 1

A computer-implemented method, comprising:
acquiring an image which is captured in the context of a microscopy measurement and images a sample to be examined;
automated monitoring of the microscopy measurement;
on the basis of the automated monitoring, creating one or more labels, wherein said one or more labels indicate semantic context information for the microscopy measurement;
on the basis of the image as input and one or more labels as ground truth, training a machine-learned algorithm which provides semantic context information for images captured in the context of microscopy measurements,
acquiring a further image, which is captured in the context of the microscopy measurement or a further microscopy measurement by the microscope and images the sample or a further sample, and
applying the trained machine-learned algorithm to the further image in order to predict further labels indicating semantic context information for the further image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein said automated monitoring of the microscopy measurement comprises automated monitoring of a user interaction with the microscope and/or the sample in the context of the microscopy measurement, and wherein said one or more labels are created on the basis of the user interaction.

### Claim 3 (depends on 1)

The method according to claim 1, wherein said one or more labels are associated with the one or more partial regions of the sample.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the user interaction relates to one or more partial regions of the sample.

### Claim 5 (depends on 3)

The method according to claim 3, wherein said one or more labels are determined on the basis of a relative imaging frequency or imaging duration of the different partial regions during the microscopy measurement.

### Claim 6 (depends on 3)

The method according to claim 3, wherein said one or more labels are determined on the basis of user-initiated analysis operations regarding properties of the sample for said one or more partial regions.

### Claim 7 (depends on 3)

The method according to claim 3, wherein said one or more labels are determined on the basis of user-initiated measurement protocol steps of the microscopy measurement for said one or more partial regions.

### Claim 8 (depends on 3)

The method according to claim 3, wherein said one or more labels are determined on the basis of user-initiated setting changes of imaging parameters of the microscope during imaging of said one or more partial regions.

### Claim 9 (depends on 3)

The method according to claim 3, wherein said one or more partial regions are determined on the basis of a field of view of the microscope.

### Claim 10 (depends on 3)

The method according to claim 3, wherein said one or more partial regions are determined on the basis of a correction of a segmentation of an analysis operation of a microscopy image.

### Claim 11 (depends on 1)

The method according to claim 1, wherein said automated monitoring of the microscopy measurement comprises automated monitoring of an automated analysis operation, and wherein said one or more labels are determined on the basis of at least one of a result of the automated analysis operation, or on correction of the analysis result of the automated analysis operation.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the automated analysis operation is implemented by the machine-learned algorithm in an earlier training state before training.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the further semantic context information indicates a global property of the sample or of the further sample.

### Claim 14 (depends on 1)

The method according to claim 1, wherein the further semantic context information indicates an image quality of the microscopy measurement.

### Claim 15 (depends on 1)

The method according to claim 1, wherein the further semantic context information indicates a degree of cleanness of the sample or of the further sample.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the further semantic context information indicates a sample carrier type.

### Claim 17 (depends on 1)

The method according to claim 1, wherein the further semantic context information indicates a suitable position for a further user interaction to be carried out.

### Claim 18 (depends on 1)

The method according to claim 1, wherein training the machine-learned algorithm is furthermore based on a further image as input and one or more further labels as ground truth, which are associated with an earlier microscopy measurement.

### Claim 19 (depends on 1)

The method according to claim 1, wherein the method furthermore comprises:
transmitting the trained machine-learned algorithm to a central server.

### Claim 20 (depends on 1)

The method according to claim 1, wherein said automated monitoring of the microscopy measurement comprises generating monitoring data indicating a temporal progression of the microscopy measurement, wherein the method furthermore comprises:
determining said one or more labels on the basis of the monitoring data.

### Claim 21 (depends on 19)

The method according to claim 19, wherein determining said one or more labels on the basis of the monitoring data is dependent on a time dependence of the monitoring data and/or is dependent on monitoring data relating to different characteristics of the microscopy measurement.

### Claim 22

A Data processing device for a microscope, comprising a computing unit and a storage unit, wherein the storage unit stores instructions that are executable by the computing unit, and wherein the microscope is configured to carry out the following steps upon the execution of the instructions in the computing unit:
acquiring an image which is captured in the context of a microscopy measurement and images a sample to be examined;
automated monitoring of the microscopy measurement;
on the basis of the automated monitoring, creating one or more labels, wherein said one or more labels indicate semantic context information for the microscopy measurement;
on the basis of the image as input and one or more labels as ground truth, training a machine-learned algorithm which provides semantic context information for images captured in the context of microscopy measurements,
acquiring a further image, which is captured in the context of the microscopy measurement or a further microscopy measurement by the microscope and images the sample or a further sample, and
applying the trained machine-learned algorithm to the further image in order to predict further semantic context information for the further image.

### Claim 23 (depends on 1)

A computer program embodied on a non-transitory computer-readable medium, comprising instructions which, upon the execution of the program by a processor, cause the latter to carry out the steps of claim 1.

### Claim 24

(canceled)
