# US 12,008,750 B2 · Generating annotation data of tissue images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,750 B2 - Document ID: US12008750B2 - Status: Granted - Kind: B2 - Application number: 17/267,846 - Filed: 2019-08-14 - Published: 2021-10-07 - Granted: 2024-06-11 - Earliest claimed: 2018-08-15 - Assignees: Koninklijke Philips N.V. - Applicants: Koninklijke Philips N.V. - Inventors: Fei Zuo; Anke Pierik; Reinhold Wimberger-Friedl; Koen De Laat; Marc Van Driel - Web page: https://patentgrep.com/patent/US12008750B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G01N 1/30: Staining; Impregnating {; Fixation; Dehydration; Multistep processes for preparing samples of tissue, cell or nucleic acid material and the like for analysis} - G01N 1/34: Purifying; Cleaning {(processes or apparatus for extracting or separating nucleic acids from biological samples C12N15/1003)} - G06T 7/11: Region-based segmentation - G06T 11/00: Two-dimensional [2D] image generation - G06T 2207/10056: Microscopic image - G06T 2207/20072: Graph-based image processing - G06T 2207/20081: Training; Learning - G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro - G06T 2207/30168: Image quality inspection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) ## Abstract Currently, there is interest in applying machine learning techniques to analyse digital pathology images automatically. Machine learning techniques often rely on training with a ground-truth image input. The quality and amount of training data determines the quality of the detector, as expressed in the rate of true and false positives, and robustness against variations in the appearance of the input images. The present application proposes to obtain image data of the same sample before and after at least one re-staining step (firstly with a structure-revealing stain, and secondly with a bio marker revealing stain). Sections of the first and second image data having a good registration relationship are chosen, along with the probability of detecting a desired candidate object (such as nucleus) and the probability of the bio marker revealing stain being present annotation data suitable for training a machine learning algorithm on the first and/or the second image data is provided. ## Claims ### Claim 1 An apparatus for automatically generating annotation data of tissue images comprising: a processor a memory storing instructions, wherein the instructions when executed by the processor cause the processor to: receive first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure, and receive second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample; and detect a plurality of candidate objects in the first image data; and obtain a corresponding plurality of detection probabilities of the detected candidate objects, generate stain confidence data of the second image data representing a likelihood that a corresponding portion of the tissue sample comprises the biomarker targeted by the second stain, and to generate image annotation data comprising a representation of the plurality of candidate objects, wherein the individual candidate objects of the plurality of candidate objects are annotated to indicate biomarker presence in corresponding portions of the second image data if an individual candidate object reaches a combined confidence based on the corresponding plurality of detection probabilities of the detected candidate objects, the corresponding stain confidence data, and a corresponding registration quality threshold of a registration between the first and second image data of the candidate objects; segment the plurality of candidate objects in the first image data, transfer the segmentation of the plurality of candidate objects in the first image data to the second image data based on the registration between the first and second image data, and sum the corresponding stain confidence data in the location of the plurality of candidate objects in the second image data, wherein the summed stain confidence data is useable with the plurality of detection probabilities of the detected candidate objects to determine a likelihood score value for candidate objects to be positive for the biomarker targeted by the second stain; and generate a registration quality map between the first image data and the second image data, wherein the processor is configured to only annotate candidate objects of the plurality of candidate objects to indicate biomarker presence when the registration quality map indicates that the registration between corresponding locations of the candidate object in the first and second image data surpasses a registration quality threshold. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the processor is further caused to generate the stain confidence data by generating a histogram of stain confidence using the second image data, and providing a plurality of stain confidence results corresponding to locations of the plurality of candidate objects in the first image data as the corresponding stain confidence data. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the processor is further caused to generate the image annotation data of the individual candidate objects of the plurality of candidate objects comprising a positive annotation of a first candidate object and a negative annotation of a second candidate object in the plurality of candidate objects. ### Claim 4 (depends on 1) The apparatus according to claim 1, wherein the processor is further caused to: generate the stain confidence data by generating a first channel of the second image data to function as a positive control, generating a second channel of the second image data to function as a negative control, and generate a histogram of stain confidence comprising generating a first histogram channel using the first channel of the second image data, to generate a second histogram channel using the second channel of the second image data, and to generate the histogram of stain confidence using the first and second histogram channels. ### Claim 5 (depends on 1) The apparatus according to claim 1, wherein the processor is further caused to detect a plurality of candidate objects in the first image data using a nucleus detector, and the corresponding plurality of detection probabilities of the detected candidate objects. ### Claim 6 (depends on 1) The apparatus of claim 1, further comprising: a graphical user interface to display the image annotation data, optionally in combination with the first and/or second image data. ### Claim 7 (depends on 6) The apparatus of claim 6, further comprising: a display configured to display the graphical user interface. ### Claim 8 A computer-implemented medical image annotation method for automatically generating annotation data of tissue images comprising: receiving first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure; receiving second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample; detecting a plurality of candidate objects in the first image data, and a corresponding plurality of detection probabilities of the detected candidate objects; generating stain confidence data of the second image data representing a likelihood that a corresponding portion of the tissue sample comprises the biomarker targeted by the second stain, comprising segmenting the plurality of candidate objects in the first image data, transferring the segmentation of the plurality of candidate objects in the first image data to the second image data based on the registration between the first and second image data, and summing the corresponding stain confidence data in the location of the plurality of candidate objects in the second image data, wherein the summed stain confidence data is used with the plurality of detection probabilities of the detected candidate objects to determine a likelihood score value for candidate objects to be positive for the biomarker targeted by the second stain; and generating image annotation data comprising a representation of the plurality of candidate objects, wherein the individual candidate objects of the plurality of candidate objects are annotated to indicate biomarker presence in corresponding portions of the second image data if an individual candidate object reaches a combined confidence based on the corresponding plurality of detection probabilities of the detected candidate objects, the corresponding stain confidence data, and a corresponding registration quality threshold of a registration between the first and second image data of the candidate objects, and wherein generating the image annotation data comprises generating a registration quality map between the first image data and the second image data, and only annotating candidate objects of the plurality of candidate objects to indicate biomarker presence when the registration quality map indicates that the registration between corresponding locations of the candidate object in the first and second image data surpasses a registration quality threshold. ### Claim 9 (depends on 8) A method for generating annotation data of tissue images comprising: acquiring a tissue sample of a patient; performing a first staining protocol on the tissue sample using a first stain for indicating tissue structure; acquiring first image data of the tissue sample using a digital microscopy imager; performing a washing protocol of the tissue sample to substantially remove the first stain; performing a second staining protocol on the tissue sample using a second stain for revealing the presence of a biomarker in the tissue sample; and acquiring second image data of the tissue sample using the digital microscopy imager; and performing the computer-implemented medical image annotation method for automatically generating annotation data of tissue images according to claim 8. ### Claim 10 (depends on 8) A non-transitory computer readable medium having stored thereon a computer program that when being executed by a processor causes the processor to perform the computer-implemented method of claim 8. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the first stain is a haematoxylin and eosin (H&E) stain, and the second stain is an immunohistochemistry (IHC) stain. ### Claim 12 (depends on 8) The computer-implemented medical image annotation method of claim 8, further comprising: a) acquiring the first image data; and b) acquiring the second image data. ### Claim 13 A method for generating annotation data of tissue images comprising: segmenting a plurality of candidate objects in first image data, wherein the first image data comprises a representation of a tissue sample stained with a first stain for indicating tissue structure; transferring the segmentation to second image data based on a registration between the first and second image data, wherein the second image data comprises a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample; summing corresponding stain confidence data in the location of the plurality of candidate objects in the second image data, wherein the summed stain confidence data is useable with a plurality of detection probabilities of detected candidate objects to determine a likelihood score value for candidate objects to be positive for the biomarker targeted by the second stain; and generating a registration quality map between the first image data and the second image data, wherein candidate objects of the plurality of candidate objects are annotated to indicate biomarker presence depending on whether the registration quality map indicates that the registration between corresponding locations of the candidate object in the first and second image data surpasses a registration quality threshold. ### Claim 14 (depends on 13) The method of claim 13, further comprising: detecting a plurality of candidate objects in the first image data, and obtaining a corresponding plurality of detection probabilities of the detected candidate objects. ### Claim 15 (depends on 13) A method for training a machine learning model comprising: receiving image annotation data generated according to the method recited in claim 13; training a machine learning model using the image annotation data; and outputting the trained machine learning model. ### Claim 16 (depends on 15) A method comprising applying a machine learning model outputted according to claim 15. ### Claim 17 (depends on 13) A non-transitory computer readable medium having stored thereon a computer program that when being executed by a processor causes the processor to perform the computer-implemented method of claim 13. ### Claim 18 (depends on 13) The computer-implemented method of claim 13, wherein the first stain is a haematoxylin and eosin (H&E) stain, and the second stain is a immunohistochemistry (IHC) stain. ### Claim 19 (depends on 13) A method for training a machine learning model comprising: acquiring image annotation data generated according to the computer-implemented method recited in claim 13; training a machine learning model using the image annotation data; and outputting the trained machine learning model.