# US 12,112,474 B2 · Image extraction device, image extraction system, image extraction method, and image extraction program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,474 B2 - Document ID: US12112474B2 - Status: Granted - Kind: B2 - Application number: 17/620,328 - Filed: 2020-06-17 - Published: 2022-08-04 - Granted: 2024-10-08 - Earliest claimed: 2019-06-19 - Assignees: The University of Tokyo - Applicants: The University of Tokyo - Inventors: Shumpei Ishikawa; Daisuke Komura; Akihiro Kawabe - Web page: https://patentgrep.com/patent/US12112474B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06T 7/0014: {using an image reference approach} - G06V 10/82: using neural networks - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - G16H 30/40: for processing medical images, e.g. editing - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G06T 2207/10056: Microscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract An image extraction device, an image extraction system, an image extraction method, and an image extraction program are capable of extracting an appropriate reference image from a database by using a pathological image as a query. The image extraction device includes: an acquisition unit which acquires a pathological image; a first calculation unit which calculates a local feature amount of the pathological image; a second calculation unit which calculates a non-local feature amount based on a correlation with the local feature amount; a third calculation unit which calculates a degree of similarity between the non-local feature amount and a feature amount of each of a plurality of reference images stored in a database; and an extraction unit which extracts one or more reference images based on the degree of similarity. ## Claims ### Claim 1 An image extraction device comprising: an acquisition unit which acquires a pathological image; a first calculation unit which calculates a local feature amount of the pathological image; a second calculation unit which calculates a non-local feature amount by performing calculation processing to reduce a dimension of a correlation matrix of the local feature amount; a third calculation unit which calculates a degree of similarity between the non-local feature amount and a feature amount of each of a plurality of reference images stored in a database; and an extraction unit which extracts one or more reference images based on the degree of similarity. ### Claim 2 (depends on 1) The image extraction device according to claim 1, further comprising a display unit which displays at least either one of the extracted one or more reference images, identification information of the one or more reference images, a type of pathology related to the one or more reference images, and the degree of similarity between the one or more reference images and the pathological image. ### Claim 3 (depends on 1) The image extraction device according to claim 1, wherein the first calculation unit calculates the local feature amount using a feature map calculated by a hidden layer included in a pre-trained convolutional neural network. ### Claim 4 (depends on 1) The image extraction device according to claim 1, wherein the second calculation unit calculates the non-local feature amount based on a correlation with a feature map calculated by a hidden layer included in a pre-trained convolutional neural network. ### Claim 5 (depends on 1) The image extraction device according to claim 1, wherein the second calculation unit calculates the non-local feature amount based on a correlation between two kinds of feature maps calculated by different hidden layers included in a pre-trained convolutional neural network. ### Claim 6 (depends on 1) The image extraction device according to claim 1, wherein the second calculation unit calculates the non-local feature amount based on a correlation between two kinds of feature maps calculated by hidden layers included in two kinds of pre-trained convolutional neural networks. ### Claim 7 (depends on 1) The image extraction device according to claim 1, wherein the plurality of reference images includes images cut out at random from regions selected in a whole slide image of pathological tissue. ### Claim 8 (depends on 1) The image extraction device according to claim 1, further comprising a determination unit which determines whether or not a certain gene mutation occurs in a cell captured in the pathological image based on the non-local feature amount. ### Claim 9 (depends on 8) The image extraction device according to claim 8, further comprising a generation unit which uses, as training data, a first image of a certain pathology, in which a cell with a certain gene mutation occurring therein is captured, and a second image of the certain pathology, in which cells without occurrence of the certain gene mutation are captured, to generate a determination model used to determine whether or not the certain gene mutation occurs in a cell captured in the pathological image based on the non-local feature amount. ### Claim 10 (depends on 9) The image extraction device according to claim 9, further comprising a display unit which, when it is determined by the determination model that the certain gene mutation occurs, displays a region of the pathological image as the basis of the determination. ### Claim 11 An image extraction system including an image extraction device and a user terminal, wherein the user terminal has an imaging unit for taking a pathological image, and the image extraction device has: an acquisition unit which acquires the pathological image; a first calculation unit which calculates a local feature amount of the pathological image; a second calculation unit which calculates a non-local feature amount by performing calculation processing to reduce a dimension of a correlation matrix of the local feature amount; a third calculation unit which calculates a degree of similarity between the non-local feature amount and a feature amount of each of a plurality of reference images stored in a database; and an extraction unit which extracts one or more reference images based on the degree of similarity. ### Claim 12 (depends on 11) The image extraction system according to claim 11, wherein the user terminal further has a display unit which displays at least either one of the extracted one or more reference images, identification information of the one or more reference images, a type of pathology related to the one or more reference images, and the degree of similarity between the one or more reference images and the pathological image. ### Claim 13 An image extraction method comprising: acquiring a pathological image; calculating a local feature amount of the pathological image; calculating a non-local feature amount by performing calculation processing to reduce a dimension of a correlation matrix of the local feature amount; calculating a degree of similarity between the non-local feature amount and a feature amount of each of a plurality of reference images stored in a database; and extracting one or more reference images based on the degree of similarity. ### Claim 14 A non-transitory computer readable medium storing an image extraction program causing an arithmetic unit included in an image extraction device to function as: an acquisition unit which acquires a pathological image; a first calculation unit which calculates a local feature amount of the pathological image; a second calculation unit which calculates a non-local feature amount by performing calculation processing to reduce a dimension of a correlation matrix of the local feature amount; a third calculation unit which calculates a degree of similarity between the non-local feature amount and a feature amount of each of a plurality of reference images stored in a database; and an extraction unit which extracts one or more reference images based on the degree of similarity.