# US 2019/0294930 A1 · Information processing device, image processing device, microscope, information processing method, and information processing program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0294930 A1
- Document ID: US20190294930A1
- Status: Published application
- Kind: A1
- Application number: 16/440,539
- Filed: 2019-06-13
- Published: 2019-09-26
- Earliest claimed: 2016-12-21
- Assignees: Nikon Corporation
- Applicants: Nikon Corporation
- Inventors: Tetsuya KOIKE; Yutaka SASAKI; Wataru TOMOSUGI; Naoya OTANI
- Web page: https://patentgrep.com/patent/US20190294930A1

## CPC

- G06V 10/82: using neural networks
- G01N 21/64: Fluorescence; Phosphorescence
- G01N 21/6458: {Fluorescence microscopy (fluorescence microscopes per se G02B21/0076 and G02B21/16)}
- G02B 21/36: arranged for photographic purposes or projection purposes (G02B21/18 takes precedence){or digital imaging or video purposes including associated control and data processing arrangements (image data processing per se G06T)}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/0499: Feedforward networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 7/00: Image analysis
- G06T 7/66: of image moments or centre of gravity
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An information processing device includes a machine learner that, in a neural network having an input layer to which data representing an image of fluorescence is input and an output layer that outputs a feature quantity of the image of fluorescence, calculates a coupling coefficient between the input layer and the output layer, using an output value that is output from the output layer when input value teacher data is input to the input layer, and feature quantity teacher data.

## Claims

### Claim 1

An information processing device comprising
a machine learner that performs machine learning by a neural network having an input layer to which data representing an image of fluorescence is input, and an output layer that outputs a feature quantity of the image of fluorescence, wherein
a coupling coefficient between the input layer and the output layer is calculated on the basis of an output value that is output from the output layer when input value teacher data is input to the input layer, and feature quantity teacher data.

### Claim 2 (depends on 1)

The information processing device according to claim 1, wherein:
the neural network includes an intermediate layer between the input layer and the output layer; and
the machine learner calculates a bias to be assigned to a neuron of the intermediate layer, using the output value that is output from the output layer when the input value teacher data is input to the input layer, and the feature quantity teacher data.

### Claim 3 (depends on 1)

The information processing device according to claim 1, the information processing device comprising
a teacher data generator that generates the input value teacher data and the feature quantity teacher data, on the basis of a predetermined point spread function.

### Claim 4 (depends on 1)

The information processing device according to claim 1, wherein:
the teacher data generator includes a centroid calculator that calculates a centroid of the image of fluorescence, using the predetermined point spread function with respect to an input image including the image of fluorescence; and
the machine learner uses the input image for the input value teacher data, and uses the centroid calculated by the centroid calculator for the feature quantity teacher data.

### Claim 5 (depends on 4)

The information processing device according to claim 4, the information processing device comprising
an extractor that extracts, from the input image, a luminance distribution of a region including the centroid calculated by the centroid calculator, wherein
the machine learner uses the luminance distribution for the input value teacher data.

### Claim 6 (depends on 4)

The information processing device according to claim 4, wherein
the teacher data generator includes:
a residual calculator that calculates a residual at a time of fitting a candidate of the image of fluorescence included in the input image to the predetermined point spread function; and
a candidate determiner that determines whether or not to use the candidate of the image of fluorescence for the input value teacher data and the feature quantity teacher data, on the basis of the residual calculated by the residual calculator.

### Claim 7 (depends on 4)

The information processing device according to claim 4, wherein:
the teacher data generator includes an input value generator that generates the input value teacher data, using the predetermined point spread function with respect to a specified centroid; and
the machine learner uses the specified centroid as the feature quantity teacher data.

### Claim 8 (depends on 7)

The information processing device according to claim 7, wherein
the input value generator combines a first luminance distribution generated using the predetermined point spread function with respect to the specified centroid with a second luminance distribution different from the first luminance distribution, to thereby generate the input value teacher data.

### Claim 9 (depends on 1)

An image processing device that calculates the feature quantity from an image obtained by image-capturing a sample containing a fluorescent substance, by a neural network using a calculation result of the machine learner output from the information processing device according to claim 1.

### Claim 10 (depends on 8)

A microscope comprising:
an image capturing device that image-captures a sample containing a fluorescent substance; and
the image processing device according to claim 8 that calculates a feature quantity of an image of fluorescence in an image that is image-captured by the image capturing device.

### Claim 11 (depends on 1)

A microscope comprising:
the information processing device according to claim 1;
an image capturing device that image-captures a sample containing a fluorescent substance; and
an image processing device that calculates a feature quantity of an image of fluorescence in an image image-captured by the image capturing device, by the neural network using the calculation result of the machine learner output from the information processing device.

### Claim 12 (depends on 10)

The microscope according to claim 10, wherein:
the fluorescent substance is activated upon receiving activation light, and emits fluorescence upon receiving excitation light in a state of being activated;
the image capturing device repeatedly image-captures the sample to obtain a plurality of first images; and
the image processing device generates a second image, using the feature quantity calculated for at least a part of the plurality of first images.

### Claim 13 (depends on 11)

The microscope according to claim 11, wherein:
the fluorescent substance is activated upon receiving activation light, and emits fluorescence upon receiving excitation light in a state of being activated;
the image capturing device repeatedly image-captures the sample to obtain a plurality of first images; and
the image processing device generates a second image, using the feature quantity calculated for at least a part of the plurality of first images.

### Claim 14 (depends on 1)

An information processing method comprising
calculating the coupling coefficient, using the information processing device according to claim 1.

### Claim 15

A non-transitory computer-readable medium storing information processing program that causes a computer to
cause a machine learner that performs machine learning by a neural network having an input layer to which data representing an image of fluorescence is input, and an output layer that outputs a feature quantity of the image of fluorescence, to perform a process of calculating a coupling coefficient between the input layer and the output layer, using an output value that is output from the output layer when input value teacher data is input to the input layer, and feature quantity teacher data.
