# US 11,442,262 B2 · Image processing device, microscope system, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,262 B2 - Document ID: US11442262B2 - Status: Granted - Kind: B2 - Application number: 16/478,434 - Filed: 2017-01-16 - Published: 2020-07-09 - Granted: 2022-09-13 - Earliest claimed: 2017-01-16 - Assignees: Nikon Corporation - Applicants: Nikon Corporation - Inventors: Ichiro Sase; Yutaka Sasaki; Takaaki Okamoto; Yuki Terui; Kohki Konishi; Masafumi Mimura; Martin Berger; Petr Gazak; Miroslav Svoboda - Web page: https://patentgrep.com/patent/US11442262B2 ## CPC - G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)} - G01N 21/6458: {Fluorescence microscopy (fluorescence microscopes per se G02B21/0076 and G02B21/16)} - G02B 21/367: {providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/37: using transform domain methods - H01J 37/222: {Image processing arrangements associated with the tube} - H01J 37/26: Electron or ion microscopes; Electron or ion diffraction tubes - H01J 37/28: with scanning beams {(H01J37/268, H01J37/292, H01J37/2955 take precedence)} - G02B 21/0024: {Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders} - G02B 21/0076: {arrangements using fluorescence or luminescence} - G02B 27/58: Optics for apodization or superresolution; Optical synthetic aperture systems - G06T 2207/10056: Microscopic image - G06T 2207/20048: Transform domain processing - G06T 2207/20221: Image fusion; Image merging - H01J 2237/226: Image reconstruction - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H01J: ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS (spark-gaps H01T; electric arc lamps H05B31/00; particle accelerators H05H) ## Abstract An image processor includes an image generator configured to generate corresponding image data corresponding to first microscopic image data obtained under a first observation condition, based on second microscopic image data and third microscopic image data obtained under a second observation condition, and an image output unit configured to output the corresponding image data. The corresponding image data may be image data corresponding to a first focal plane from which the first microscopic image data are obtained, and wherein the second microscopic image data and the third microscopic image data may be image data on a second focal plane and a third focal plane, respectively, which are different from the first focal plane. ## Claims ### Claim 1 A microscope system, comprising: a first microscope using a first microscopy; a second microscope using a second microscopy that is different from the first microscopy; and an image processor comprising: an obtaining unit configured to obtain first microscopic image data obtained by the first microscopy and second microscopic image data obtained by the second microscopy that is different from the first microscopy; and an image generator configured to generate, based on the second microscopic image data, corresponding image data of the second microscopy corresponding to the first microscopic image data, wherein the first microscopic image data and the second microscopic image data are obtained from the same observation position of the first and second microscopes. ### Claim 2 (depends on 1) The microscope system according to claim 1, wherein the corresponding image data is image data corresponding to a first plane from which the first microscopic image data is obtained, and the second microscopic image data, which is not obtained from the first plane, is image data on a second plane and a third plane, the second and third planes being different from the first plane. ### Claim 3 (depends on 2) The microscope system according to claim 2, wherein the image generator is configured to generate the corresponding image data by performing Fourier transform and inverse Fourier transform on the second microscopic image data on the second plane and the third plane. ### Claim 4 (depends on 2) The microscope system according to claim 2, wherein the first plane is a plane between the second plane and the third plane. ### Claim 5 (depends on 2) The microscope system according to claim 2, wherein the second microscopy is a localization method, and the image generator is configured to generate the corresponding image data by extracting image data within a focal depth of the first microscope using the first microscopy from the second microscopic image data. ### Claim 6 (depends on 2) The microscope system according to claim 2, wherein the image generator is configured to generate the corresponding image data based on (i) a first pattern in an image that is based on the second microscopic image data on the second plane and (ii) a second pattern in an image that is based on the second microscopic image data on the third plane, the second pattern corresponding to the first pattern. ### Claim 7 (depends on 1) The microscope system according to claim 1, wherein the corresponding image data is image data corresponding to a first time at which the first microscopic image data is obtained, and the second microscopic image data is image data corresponding to a second time and to a third time, which are different from the first time. ### Claim 8 (depends on 7) The microscope system according to claim 7, wherein the first time is between the second time and the third time. ### Claim 9 (depends on 7) The microscope system according to claim 7, wherein the image generator is configured to generate the corresponding image data based on (i) a first pattern in an image that is based on the second microscopic image data obtained at the second time and (ii) a second pattern in an image that is based on the second microscopic image data obtained at the third time. ### Claim 10 (depends on 1) The microscope system according to claim 1, further comprising an evaluator configured to perform an evaluation on the corresponding image data based on an observation condition, in the second microscopy, of the second microscopic image data. ### Claim 11 (depends on 1) The microscope system according to claim 1, further comprising: a display; and an input unit. ### Claim 12 (depends on 1) The microscope system according to claim 1, wherein the image generator is configured to generate, based on the second microscopic image data, the corresponding image data corresponding to a position at which the first microscopic image data is obtained. ### Claim 13 (depends on 2) The microscope system according to claim 2, wherein the image generator is configured to generate, based on the second microscopic image data, the corresponding image data based on a distance between the first plane and the second plane, a distance between the first plane and the third plane, and the second microscopic image data. ### Claim 14 (depends on 1) The microscope system according to claim 1, wherein the image generator is configured to generate, based on the second microscopic image data, the corresponding image data corresponding to a focal depth of the first microscope using the first microscopy. ### Claim 15 (depends on 1) The microscope system according to claim 1, wherein the image generator is configured to generate, based on the second microscopic image data, the corresponding image data corresponding to a time at which the first microscopic image data are obtained. ### Claim 16 (depends on 9) The microscope system according to claim 9, wherein the image generator is configured to generate the corresponding image data based on (i) a position of the first pattern, (ii) an intensity of the first pattern, (iii) a position of the second pattern, and (iv) an intensity of the second pattern, the first pattern corresponding to the second pattern. ### Claim 17 (depends on 9) The microscope system according to claim 9, wherein the image generator is configured to generate the corresponding image data based on an intensity of the first pattern and an intensity of the second pattern. ### Claim 18 (depends on 1) The microscope system according to claim 1, wherein the corresponding image data is image data corresponding to the second microscopic image data that has not been obtained. ### Claim 19 (depends on 1) The microscope system according to claim 1, further comprising an image output unit configured to output the corresponding image data or the first microscopic image data to a display such that the display displays an image based on the corresponding image data or both the image based on the corresponding image data and an image based on the first microscopic image data. ### Claim 20 (depends on 19) The microscope system according to claim 19, wherein the image output unit is configured to output the corresponding image data such that the image based on the corresponding image data is visually distinguishable from an image based on the second microscopic image data. ### Claim 21 (depends on 1) The processor microscope system according to claim 1, wherein the image generator is configured to: determine whether the second microscopic image data corresponding to the first microscopic image data exists based on an observation condition of the first microscopy and an observation condition of the second microscopy, and generate the corresponding image data in a case that the image generator has determined that the second microscopic image data does not exist. ### Claim 22 (depends on 19) The microscope system according to claim 19, wherein the image output unit is configured to output the first microscopic image data and the second microscopic image data to the display such that the display displays the image based on the first microscopic image data and an image based on the second microscopic image data, and the image processor further comprises an accepting unit configured to accept an instruction of generating the corresponding image data via an input unit. ### Claim 23 (depends on 10) The microscope system according to claim 10, wherein the evaluator is configured to determine, based on the observation condition of the second microscopic image data in the second microscopy, whether to generate the corresponding image data or obtain the second microscopic image data corresponding to the first microscopic image data by using the second microscopy. ### Claim 24 (depends on 19) The microscope system according to claim 19, further comprising an evaluator configured to perform an evaluation on the corresponding image data based on an observation condition, in the second microscopy, of the second microscopic image data, the image output unit being configured to output a result of the evaluation on the corresponding image data to a display such that the display displays the result of the evaluation on the corresponding image data, and the image processor further comprises an accepting unit configured to accept, via an input unit, an instruction of generating the corresponding image data or an instruction of obtaining the second microscopic image data corresponding to the first microscopic image data by using the second microscopy. ### Claim 25 An image processing method comprising: obtaining (i) first microscopic image data by a first microscopy used by a first microscope and (ii) second microscopic image data by a second microscopy that is different from the first microscopy and is used by a second microscope; and generating, based on the second microscopic image data and third microscopic image data, corresponding image data of the second microscopy corresponding to the first microscopic image data, wherein the first microscopic image data and the second microscopic image data are obtained from the same observation position of the first and second microscopes. ### Claim 26 (depends on 25) The image processing method according to claim 25, wherein the corresponding image data is image data corresponding to a first plane from which the first microscopic image data is obtained, and the second microscopic image data, which is not obtained from the first plane, is image data on a second plane and a third plane, which are different from the first plane. ### Claim 27 (depends on 26) The image processing method according to claim 26, wherein, in a case that generating the corresponding image data is performed, Fourier transform and inverse Fourier transform are performed on the second microscopic image data on the second plane and the third plane to generate the corresponding image data. ### Claim 28 (depends on 26) The image processing method according to claim 26, wherein the first plane is a plane between the second plane and the third plane. ### Claim 29 (depends on 26) The image processing method according to claim 26, wherein the second microscopy is a localization method, and in a case that generating the corresponding image data is performed, the corresponding image data is generated by extracting image data within a focal depth of the first microscope using the first microscopy from the second microscopic image data. ### Claim 30 (depends on 26) The image processing method according to claim 26, wherein in a case that generating the corresponding image data is performed, the corresponding image data is generated based on (i) a first pattern in an image based on the second microscopic image data on the second plane and (ii) a second pattern in an image based on the second microscopic image data on the third plane, the second pattern corresponding to the first pattern. ### Claim 31 (depends on 25) The image processing method according to claim 25, wherein the corresponding image data is image data corresponding to a first time at which the first microscopic image data is obtained, and the second microscopic image data is image data corresponding to a second time and to a third time, which are different from the first time. ### Claim 32 (depends on 31) The image processing method according to claim 31, wherein the first time is between the second time and the third time. ### Claim 33 (depends on 31) The image processing method according to claim 31, wherein in a case that generating the corresponding image data is performed, the corresponding image data is generated based on (i) a pattern in an image based on the second microscopic image data obtained at the second time and (ii) a pattern in an image based on the second microscopic image data obtained at the third time. ### Claim 34 (depends on 25) The image processing method according to claim 25, further comprising evaluating the corresponding image data based on an observation condition, in the second microscopy, of the second microscopic image data. ### Claim 35 (depends on 25) The image processing method according to claim 25, further comprising outputting the corresponding image data or the first microscopic image data to a display such that the display displays an image based on the corresponding image data or both the image based on the corresponding image data and an image based on the first microscopic image data. ### Claim 36 A non-transitory computer-readable medium storing a program for causing a computer to: obtain (i) first microscopic image data by a first microscopy used by a first microscope and (ii) second microscopic image data by a second microscopy that is different from the first microscopy and is used by a second microscope; and generate, based on the second microscopic image data, corresponding image data of the second microscopy corresponding to the first microscopic image, wherein the first microscopic image data and the second microscopic image data are obtained from the same observation position of the first and second microscopes. ### Claim 37 (depends on 36) The non-transitory computer-readable medium storing the program according to claim 36, wherein the program further causes the computer to output the corresponding image data or the first microscopic image data to a display such that the display displays an image based on the corresponding image data or both the image based on the corresponding image data and an image based on the first microscopic image data.