# US 2021/0174147 A1 · Operating method of image processing apparatus, image processing apparatus, and computer-readable recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174147 A1
- Document ID: US20210174147A1
- Status: Published application
- Kind: A1
- Application number: 17/182,643
- Filed: 2021-02-23
- Published: 2021-06-10
- Earliest claimed: 2018-10-09
- Assignees: Olympus Corporation
- Applicants: Olympus Corporation
- Inventors: Takeshi OTSUKA
- Web page: https://patentgrep.com/patent/US20210174147A1

## CPC

- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2431: Multiple classes
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/90: Determination of colour characteristics
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/698: {Matching; Classification}
- G16H 10/40: for data related to laboratory analysis, e.g. patient specimen analysis
- G16H 30/40: for processing medical images, e.g. editing
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2210/41: Medical
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An operating method of an image processing apparatus includes: estimating, from an optical spectrum of each pixel of each training image of training images prepared by first specimen preparing process protocols, staining characteristics of each pigment at the pixel; recording the first specimen preparing process protocol for the training image in association with the estimated staining characteristics; estimating, from an optical spectrum of each pixel of an input training image, staining characteristics of each pigment at the pixel of the input training image prepared by a second specimen preparing process protocol; converting the staining characteristics of at least one selected pigment in the input training image into the staining characteristics of the at least one selected pigment of any one of the training images; and generating, based on the converted staining characteristics, a virtual stained specimen image that is stained by a third specimen preparing process protocol.

## Claims

### Claim 1

An operating method of an image processing apparatus, comprising:
estimating, from an optical spectrum of each pixel of each training image of plural training images, a pigment spectrum and a pigment quantity that are staining characteristics of each pigment at the pixel of the training image, the plural training images being stained specimen images prepared by plural first specimen preparing process protocols different from one another, each first specimen preparing process protocol including staining using plural pigments;
recording the first specimen preparing process protocol for the training image in association with the estimated staining characteristics;
estimating, from an optical spectrum of each pixel of an input training image, staining characteristics of each pigment at the pixel of the input training image, the input training image being a stained specimen image prepared by a second specimen preparing process protocol different from the plural first specimen preparing process protocols and including staining using the plural pigments, the input training image being input as a training image for learning;
converting the staining characteristics of at least one selected pigment in the input training image into the staining characteristics of the at least one selected pigment of any one of the plural training images; and
generating, based on the converted staining characteristics of the at least one selected pigment in the input training image, a virtual stained specimen image that is stained by a third specimen preparing process protocol different from the first specimen preparing process protocols and the second specimen preparing process protocol.

### Claim 2 (depends on 1)

The operating method according to claim 1, further comprising
repeatedly performing the converting to convert the staining characteristics of each pigment in the input training image into the recorded staining characteristics of the pigment in each training image, and
repeatedly performing the generating based on the converted staining characteristics of each pigment in the input training image.

### Claim 3 (depends on 1)

The operating method according to claim 1, further comprising
estimating, from the staining characteristics in each pixel of the plural training images and the input training image, a tissue to which the pixel of the plural training images or the input training image belongs,
converting the staining characteristics of at least one selected tissue in the input training image into the recorded staining characteristics of the at least one selected tissue of any one of the plural training images, and
generating, based on the converted staining characteristics in the tissue of the at least one selected tissue in the input training image, a virtual stained specimen image that is stained by a third specimen preparing process protocol different from the first specimen preparing process protocols and the second specimen preparing process protocol.

### Claim 4 (depends on 1)

The operating method according to claim 1, further comprising
estimating, from an optical spectrum of each pixel of an input image that is a stained specimen image including staining using the plural pigments, staining characteristics of each pigment at the pixel of the input image,
calculating, from a data set including the first specimen preparing process protocols or the second specimen preparing process protocol and a correct answer image or images, an estimation operator for obtaining a correct answer image for the input image by estimation using regression analysis or by classification, and
estimating, based on the calculated estimation operator, a correct answer image from the input image.

### Claim 5 (depends on 3)

The operating method according to claim 3, further comprising
classifying each pixel of the input training image from the staining characteristics of the pixel of the input training image,
performing classification into plural tissues according to the classification of each pixel of the input training image, and
calculating, based on the staining characteristics in pixels belonging to the classified tissues, feature data as the staining characteristics in each tissue of the classified tissues.

### Claim 6

An image processing apparatus comprising a processor comprising hardware, the processor being configured to:
estimate, from an optical spectrum of each pixel of each training image of plural training images, a pigment spectrum and a pigment quantity that are staining characteristics of each pigment at the pixel of the training image, the plural training images being stained specimen images prepared by plural first specimen preparing process protocols different from one another, each first specimen preparing process protocol including staining using plural pigments;
record the first specimen preparing process protocol for the training image in association with the estimated staining characteristics;
estimate, from an optical spectrum of each pixel of an input training image, staining characteristics of each pigment at the pixel of the input training image, the input training image being a stained specimen image prepared by a second specimen preparing process protocol different from the plural first specimen preparing process protocols and including staining using the plural pigments, the input training image being input as a training image for learning;
convert the staining characteristics of at least one selected pigment in the input training image into the staining characteristics of the at least one selected pigment of any one of the plural training images; and
generate, based on the converted staining characteristics of the at least one selected pigment in the input training image, a virtual stained specimen image that is stained by a third specimen preparing process protocol different from the first specimen preparing process protocols and the second specimen preparing process protocol.

### Claim 7

A non-transitory computer-readable recording medium with an executable program stored thereon, the program causing an image processing apparatus to execute:
estimating, from an optical spectrum of each pixel of each training image of plural training images, a pigment spectrum and a pigment quantity that are staining characteristics of each pigment at the pixel of the training image, the plural training images being stained specimen images prepared by plural first specimen preparing process protocols different from one another, each first specimen preparing process protocol including staining using plural pigments;
recording the first specimen preparing process protocol for the training image in association with the estimated staining characteristics;
estimating, from an optical spectrum of each pixel of an input training image, staining characteristics of each pigment at the pixel of the input training image, the input training image being a stained specimen image prepared by a second specimen preparing process protocol different from the plural first specimen preparing process protocols and including staining using the plural pigments, the input training image being input as a training image for learning;
converting the staining characteristics of at least one selected pigment in the input training image into the staining characteristics of the at least one selected pigment of any one of the plural training images; and
generating, based on the converted staining characteristics of the at least one selected pigment in the input training image, a virtual stained specimen image that is stained by a third specimen preparing process protocol different from the first specimen preparing process protocols and the second specimen preparing process protocol.
