# US 2021/0342636 A1 · Optimization of workflows for microscopes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342636 A1
- Document ID: US20210342636A1
- Status: Published application
- Kind: A1
- Application number: 17/285,477
- Filed: 2019-09-25
- Published: 2021-11-04
- Earliest claimed: 2018-10-18
- Assignees: Leica Microsystems CMS GmbH
- Applicants: Leica Microsystems Cms GmbH
- Inventors: Frank SIECKMANN; Constantin KAPPEL
- Web page: https://patentgrep.com/patent/US20210342636A1

## CPC

- G06V 20/698: {Matching; Classification}
- G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2431: Multiple classes
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06N 20/00: Machine learning
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/82: using neural networks
- G06V 20/693: {Acquisition}
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G05B 17/02: electric
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method for optimizing a workflow of at least one microscope or microscope system includes a step a) of implementing a workflow by one or more components of at least one microscope and/or microscope system, wherein the workflow comprises a capture of first data. In a step b), a trained model is determined for the workflow, at least in part based on the captured first data.

## Claims

### Claim 1

A method for optimizing a workflow of at least one microscope or microscope system, the method comprising:
a) implementing a workflow by one or more components of at least one microscope and/or microscope system, wherein the workflow comprises a capture of first data; and
b) determining a trained model for the workflow, at least in part based on the basis of the captured first data.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the step of determining the trained model comprises an analysis of the first data by a trained master model.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the trained master model is applied as a web service, on a workstation, on the at least one microscope or microscope system or on an attached component of the at least one microscope or microscope system.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the step of determining the trained model comprises selecting, from a multiplicity of trained models, a model that has been trained in a certain way.

### Claim 5 (depends on 4)

The method according to claim 4, wherein:
the multiplicity of trained models are classified by field of use, and/or
the multiplicity of trained models are organized hierarchically, and/or
individual trained models from the multiplicity of trained models are specialized for individual types of specimens, experiments, measurements, recording modalities or device settings.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising automatically applying the determined trained model.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the determined trained model is automatically applied to the captured first data and/or new captured data.

### Claim 8 (depends on 6)

The method according to claim 6, wherein at least one decision relating to the workflow is made based on the application of the trained model.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising adapting the trained model, wherein the adaptation of the trained model comprises at least one of the following steps:
training part of the trained model; and/or
training the trained model using aggregated data, wherein the aggregated data originate from one or more sources, and wherein the aggregated data comprise data uploaded to a cloud, a server or a workstation in automatic, semiautomatic or manual fashion.

### Claim 10 (depends on 2)

The method according to claim 2, further comprising adapting the trained master model by training part of the trained master model based on annotated data of a second class such that a prediction accuracy of the adapted trained master model is increased in comparison with a prediction accuracy of the trained master model when applied to the captured first data, wherein the trained master model was trained using data of a first class and the captured first data are data of the second class, which differs from the first class.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the trained model is determined during the course of a measurement carried out by the one or more microscopes and/or microscope systems.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the trained model was trained by deep learning on a cloud, a server or a workstation using data from one or more sources and was loaded onto the at least one microscope or microscope system and/or onto an attached component of the at least one microscope or microscope system.

### Claim 13

An apparatus for optimizing workflows of one or more microscopes and/or microscope systems, the apparatus comprising:
one or more processors;
one or more computer-readable storage media on which computer-executable instructions are stored, the instructions, when executed by the one or more processors, cause execution of the following steps:
a) implementing, by one or more components of one or more microscopes and/or microscope systems, a workflow, wherein the workflow comprises a capture of first data; and
b) determining one or more trained models for the workflow, at least in part based on the captured first data.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein the workflow comprises an application of a trained master model to the captured first data, and wherein the determination of the one or more trained models for the workflow is at least partly implemented based on the basis of the application of the trained master model to the captured first data.

### Claim 15 (depends on 14)

The apparatus according to claim 14, wherein the application of the trained master model comprises an analysis of the captured first data, and wherein the trained master model determines the one or more trained models for the workflow based on information from the analysis of the captured first data.

### Claim 16 (depends on 14)

The apparatus according to claim 14, wherein the apparatus is configured to adapt the trained master model, and wherein the adaptation comprises training of only one part or of a plurality of parts of the trained master model.

### Claim 17 (depends on 13)

The apparatus according to claim 13, wherein the computer-executable instructions, when executed by the one or more processors, further causes execution of the following step:
c) applying at least one of the determined one or more trained models to be applied to the captured first data, and wherein:
at least one decision relating to the workflow is made based on the application of the at least one trained model, and/or the at least one trained model is determined and automatically applied during an ongoing measurement.

### Claim 18 (depends on 13)

The apparatus according to claim 13, wherein the captured first data comprise at least one of the following: image data; user inputs; error messages; metadata; parameter data of the one or more components; data relating to the progress of the experiment; information relating to reagents and materials; information relating to an object or a specimen; user-related data; and device data of devices that are controlled during the course of a measurement carried out by the one or more microscopes and/or microscope systems and/or by attached components of the one or more microscopes and/or microscope systems.

### Claim 19 (depends on 13)

The apparatus according to claim 13, wherein the one or more processors comprise computing accelerators such as graphical processing units (GPUs), TensorFlow processing units (TPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs), or at least one central processing unit (CPU).

### Claim 20 (depends on 13)

A system for adapting a workflow, the system comprising:
at least one of the apparatus according to claim 13, wherein the at least one apparatus is part of the one or more microscopes and/or microscope systems and is configured to capture second data;
a cloud comprising computing resources, wherein the computing resources are configured to train one or more models; and
a model manager, which is configured to implement the one or more trained models on at least one of the at least one apparatus.

### Claim 21 (depends on 20)

The system according to claim 20, wherein:
the computing resources are configured to adapt at least one of the one or more trained models based on the captured second data,
the second data comprise at least one of the following: the at least one trained model or parts thereof; captured data which comprise the input data for the one or more trained models; annotations of the input data; hidden representations of the data; assessments of output values of the trained model applied to the input data; and user inputs, and/or
wherein the one or more trained models were trained using different data types.

### Claim 22 (depends on 20)

The system according to claim 20, further comprising a model store as a software component on the at least one apparatus, wherein the model store is set up to select trained models, and/or the system is configured to download trained models using an interface and the model store for purposes of local execution.

### Claim 23 (depends on 20)

The system according to claim 20, wherein the model manager is configured to implement the one or more trained models on the at least one apparatus based on an event, and wherein the event comprises at least one of the following:
receiving a request from a user via a web service;
receiving a request based on a result of an analysis of the first data by one of the one or more trained models;
receiving a request based on a step in the workflow;
receiving one or more results by data mining on model metadata, hidden representations or model parameters;
receiving a request from a model store; and
receiving a request based on a capture of explicit model parameters which include experimental conditions.

### Claim 24 (depends on 20)

The system according to claim 20, wherein the model manager is configured to:
regulate rights management using a model database, wherein the model database stores at least one of the following information items: a model architecture, model parameters, model metadata and model rights;
control communication between individual ones or groups of the at least one apparatus;
manage versions of trained models;
assign one of the trained models to a field of application;
output one of the trained models within a container to the at least one apparatus, wherein the model manager comprises a container manager configured to create or provide containers for executing the trained models; and/or
manage metadata of the trained models, wherein the metadata are used to manage the trained models and/or wherein the metadata comprise at least one of the following: a unique identification of the trained model, a version number of the trained model, an identification of a use location in a device, a category of the trained model, and an object of the trained model.
