# US 2022/0067444 A1 · Microscopy System and Method for Monitoring a Learning Process of a Machine Learning Model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0067444 A1
- Document ID: US20220067444A1
- Status: Published application
- Kind: A1
- Application number: 17/462,234
- Filed: 2021-08-31
- Published: 2022-03-03
- Earliest claimed: 2020-09-02
- Assignees: Carl Zeiss Microscopy GmbH
- Applicants: Carl Zeiss Microscopy GmbH
- Inventors: Manuel Amthor; Daniel Haase
- Web page: https://patentgrep.com/patent/US20220067444A1

## CPC

- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 20/00: Machine learning
- G02B 21/367: {providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/094: Adversarial learning
- G06V 10/776: Validation; Performance evaluation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A microscopy system and a method for monitoring a learning process of a machine learning model are described. The microscopy system comprises a microscope with a camera for capturing a microscope image and a computing device. The computing device processes the microscope image by means of a machine learning model. A learning process of the machine learning model is conducted with a training system. In the learning process, model parameter values of the machine learning model are adjusted using training data. During the learning process, a quality measure based on the training data and a quality measure based on validation data are calculated for respectively current model parameter values. A training learning progression and a validation learning progression are formed from the quality measures The training system comprises a verification model, which is fed with the training learning progression and validation learning progression during the learning process. The verification model is designed to generate a quality assessment of the learning process depending on the training learning progression and validation learning progression.

## Claims

### Claim 1

A microscopy system
with at least one microscope, which comprises at least one camera for capturing a microscope image and a computing device, wherein the computing device comprises an imaging processing program for processing the microscope image by means of a machine learning model; and
with a training system for carrying out a learning process of the machine learning model, wherein model parameter values of the machine learning model are adjusted in the learning process using training data, wherein during the learning process a quality measure based on the training data and a quality measure based on validation data are calculated for current model parameter values and a training learning progression and a validation learning progression are formed from the quality measures;
wherein the training system comprises a verification model, which is fed with the training learning progression and the validation learning progression during the learning process; and
wherein the verification model is configured to generate a quality assessment of the learning process of the machine learning model depending on the training learning progression and the validation learning progression.

### Claim 2

A method for monitoring a learning process of a machine learning model, comprising
launching a learning process using training data in order to adjust model parameter values of the machine learning model;
wherein during the learning process a quality measure based on the training data and a quality measure based on validation data are calculated for current model parameter values and a training learning progression and a validation learning progression are formed from the quality measures,
wherein the training learning progression and the validation learning progression are fed to a verification model during the learning process; and
wherein the verification model generates a quality assessment of the learning process of the machine learning model depending on the training learning progression and the validation learning progression.

### Claim 3 (depends on 2)

The method as defined in claim 2,
wherein the training learning progression and the validation learning progression are fed to the verification model during an ongoing training of the machine learning model before a predetermined stopping criterion of the training is reached.

### Claim 4 (depends on 3)

The method as defined in claim 3,
wherein a decision is made based on the quality assessment whether to continue or abort the ongoing training of the machine learning model.

### Claim 5 (depends on 2)

The method as defined in claim 2,
wherein the verification model comprises a verification machine learning model, which is trained to generate the quality assessment as output from the training learning progression and the validation learning progression as input data.

### Claim 6 (depends on 5)

The method as defined in claim 5,
wherein the verification machine learning model is trained by an unsupervised learning process, in which a plurality of training learning progressions and associated validation learning progressions are used as verification machine learning model training data, or
wherein the verification machine learning model is trained by a supervised learning process, in which a plurality of training learning progressions and associated validation learning progressions with a predetermined quality assessment are used as verification machine learning model training data.

### Claim 7 (depends on 5)

The method as defined in claim 5,
wherein the training learning progression and the validation learning progression are respectively fed to the verification machine learning model as a sequence of quality measure values; and
wherein the verification machine learning model comprises a recurrent neural network.

### Claim 8 (depends on 5)

The method as defined in claim 5,
wherein the training learning progression and the validation learning progression are fed to the verification machine learning model as graphs in the form of image data; and
wherein the verification machine learning model comprises a convolutional neural network.

### Claim 9 (depends on 2)

The method as defined in claim 2,
wherein the verification model takes one or more of the following factors into account for the quality assessment:
jumps in the training learning progression or validation learning progression;
number of epochs after which the training learning progression or validation learning progression saturates, and a value of the quality measures during saturation;
difference between the training learning progression and the validation learning progression;
divergence of the training learning progression or validation learning progression;
initial fluctuations in the training learning progression and validation learning progression and subsequent monotonous training learning progression and validation learning progression;
whether an optimum of the model parameter values at which a quality measure is below a predetermined limit value is reached early.

### Claim 10 (depends on 2)

The method as defined in claim 2,
wherein the quality assessment comprises a suggestion for a modification of training parameters during the ongoing learning process or for a new learning process to be initiated.

### Claim 11 (depends on 10)

The method as defined in claim 10,
wherein the modification of training parameters comprises at least one of: a modification of a learning rate and a modification of a set number of epochs.

### Claim 12 (depends on 10)

The method as defined in claim 10,
wherein, in the event that the quality assessment assumes a local optimum of the model parameter values, the modification of training parameters comprises a one-time or repeated increase of a learning rate in order to escape the local optimum of the model parameter values.

### Claim 13 (depends on 10)

The method as defined in claim 10,
wherein the modification of training parameters comprises a modification of the model parameter values;
wherein the verification machine learning model also receives to this end, in addition to the training learning progression and the validation learning progression, associated model parameter values of the machine learning model as inputs.

### Claim 14 (depends on 13)

The method as defined in claim 13,
wherein the verification machine learning model comprises a neural network trained by a supervised learning process, in which training data comprises a plurality of training learning progressions, validation learning progressions and associated model parameter values, as well as modifications of the model parameter values as target data; or
wherein the verification machine learning model comprises a neural network trained by a reinforcement learning method, in which a reinforcement learning agent learns using a predefined training environment, for an input comprising a training learning progression, a validation learning progression and associated model parameter values, how to modify the model parameter values in order to optimize the quality assessment.

### Claim 15 (depends on 2)

The method as defined in claim 2,
wherein the training learning progression and the validation learning progression are input into a prediction machine learning model that is trained to predict future progressions of an input training learning progression and validation learning progression from the input training learning progression and validation learning progression and to add the predicted future progressions onto the input training learning progression and validation learning progression,
wherein the training learning progression and validation learning progression supplemented by the prediction are output to a user or to the verification model.

### Claim 16 (depends on 2)

The method as defined in claim 2,
wherein the verification machine learning model is configured to conduct an anomaly detection in order to determine deviations from typical training progressions,
wherein the verification machine learning model for the anomaly detection is designed as an autoencoder trained with training learning progressions and validation learning progressions that do not contain any anomalies.

### Claim 17 (depends on 2)

The method as defined in claim 2,
wherein the verification model is designed to define assessment criteria for the generation of the quality assessment depending on contextual data, wherein the contextual data relate to the machine learning model or the training of the machine learning model,
wherein the contextual data comprise information regarding one or more of the following aspects:
type of a learning method, wherein a distinction is made at least between supervised training, unsupervised training, reinforcement learning and a use of adversarial networks;
type of a task of the machine learning model, wherein a distinction is made at least between a classification, segmentation, detection or regression;
architecture of the machine learning model;
values of hyperparameters of the machine learning model;
a type of the training/validation data.

### Claim 18 (depends on 2)

A computer program with commands that, when executed by a computer, cause the execution of the method defined in claim 2.

### Claim 19

A method for monitoring a learning process of a machine learning model containing a generative adversarial network, comprising
launching a learning process using training data in order to adjust model parameter values of the machine learning model;
wherein during the learning process at least one quality measure is calculated for respectively current model parameter values and at least one training learning progression is formed from the quality measures;
wherein the at least one training learning progression is fed to a verification model during the learning process; and
wherein the verification model generates a quality assessment of the learning process of the machine learning model depending on the at least one training learning progression.
