# US 10,769,788 B2 · Few-shot learning based image recognition of whole slide image at tissue level > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,788 B2 - Document ID: US10769788B2 - Status: Granted - Kind: B2 - Application number: 16/129,621 - Filed: 2018-09-12 - Published: 2019-03-14 - Granted: 2020-09-08 - Earliest claimed: 2017-09-12 - Assignees: NantOmics, LLC - Applicants: NantOmics, LLC - Inventors: Bing Song; Mustafa Jaber - Web page: https://patentgrep.com/patent/US10769788B2 ## CPC - G06T 7/0014: {using an image reference approach} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06T 7/001: {using an image reference approach} - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06T 7/73: using feature-based methods - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/698: {Matching; Classification} - G16H 10/40: for data related to laboratory analysis, e.g. patient specimen analysis - G16H 10/60: for patient-specific data, e.g. for electronic patient records - G16H 30/40: for processing medical images, e.g. editing - G16H 40/67: for remote operation - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30004: Biomedical image processing - G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro - G06T 2207/30096: Tumor; Lesion - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract A computer implemented method of generating at least one shape of a region of interest in a digital image is provided. The method includes obtaining, by an image processing engine, access to a digital tissue image of a biological sample; tiling, by the image processing engine, the digital tissue image into a collection of image patches; obtaining, by the image processing engine, a plurality of features from each patch in the collection of image patches, the plurality of features defining a patch feature vector in a multidimensional feature space including the plurality of features as dimensions; determining, by the image processing engine, a user selection of a user selected subset of patches in the collection of image patches; classifying, by applying a trained classifier to patch vectors of other patches in the collection of patches, the other patches as belonging or not belonging to a same class of interest as the user selected subset of patches; and identifying one or more regions of interest based at least in part on the results of the classifying. ## Claims ### Claim 1 A computer implemented method of generating at least one shape of a region of interest in a digital image, the method comprising: obtaining, by an image processing engine, access to a digital tissue image of a biological sample; tiling, by the image processing engine, the digital tissue image into a collection of image patches; obtaining, by the image processing engine, a plurality of features from each patch in the collection of image patches, the plurality of features defining a patch feature vector in a multidimensional feature space including the plurality of features as dimensions; determining, by the image processing engine, a user selection of a user selected subset of patches in the collection of image patches, wherein the user selection further includes a selection, prior to selection of the subset of patches in the collection of image patches, of whether the user selected subset of patches are positive or negative patches; classifying, by applying a trained classifier to patch vectors of other patches in the collection of patches, the other patches as belonging or not belonging to a same class of interest as the user selected subset of patches; and identifying one or more regions of interest based at least in part on the results of the classifying. ### Claim 2 (depends on 1) The method of claim 1, wherein the step of tiling the digital tissue image includes creating image patches that are of a uniform size and a uniform shape. ### Claim 3 (depends on 2) The method of claim 2, wherein the image patches of uniform size and uniform shape include square patches of less than or equal to 1,000 pixels by 1,000 pixels. ### Claim 4 (depends on 3) The method of claim 3, wherein the square patches are less than or equal to 400 pixels by 400 pixels. ### Claim 5 (depends on 3) The method of claim 3, wherein the square patches are less than or equal to 256 pixels by 256 pixels. ### Claim 6 (depends on 1) The method of claim 1, wherein the step of tiling the digital tissue image includes creating image patches that are of non-uniform size and shape. ### Claim 7 (depends on 1) The method of claim 1, wherein the collection of image patches includes non-overlapping patches. ### Claim 8 (depends on 1) The method of claim 1, wherein the trained classifier includes a support vector machine. ### Claim 9 (depends on 8) The method of claim 8, wherein the support vector machine includes a two-class support vector machine. ### Claim 10 (depends on 1) The method of claim 1, wherein the trained classifier comprises a support vector machine and training patches used to train the support vector machine are generated by a second support vector machine. ### Claim 11 (depends on 10) The method of claim 10, wherein the second support vector machine is a one-class support vector machine that is trained by training patches comprising the user selected subset of patches. ### Claim 12 (depends on 10) The method of claim 10, wherein the support vector machine is a two-class support vector machine and the second support vector machine is a one-class support vector machine that is trained by training patches comprising the user selected subset of patches. ### Claim 13 (depends on 1) The method of claim 1, wherein obtaining the features comprises submitting each patch to a trained image processing neural network that executes feature extraction processing, the feature extraction processing having been trained on images of a variety of known objects. ### Claim 14 (depends on 13) The method of claim 13, wherein the neural network is a convolutional neural network. ### Claim 15 (depends on 13) The method of claim 13, wherein the neural network has been trained on at least 1,000,000 images and the variety of known objects comprises objects belonging to at least 1,000 different classes of known objects. ### Claim 16 (depends on 13) The method of claim 13, wherein a majority of the known objects are not digital tissue images of biological samples. ### Claim 17 (depends on 15) The method of claim 15, wherein a majority of the known objects are not digital tissue images of biological samples. ### Claim 18 (depends on 13) The method of claim 13, wherein all of the known objects are not digital tissue images of biological samples. ### Claim 19 (depends on 13) The method of claim 13, wherein all of the known objects are not in the same class of interest as the user selected subset of patches. ### Claim 20 (depends on 1) The method of claim 1, further comprising causing a computing device to render the region of interest shapes on a display. ### Claim 21 (depends on 1) The method of claim 1, wherein the region of interest shapes includes at least one tissue mask. ### Claim 22 (depends on 21) The method of claim 21, wherein the at least one tissue mask comprises a microdissection mask. ### Claim 23 (depends on 1) The method of claim 1, wherein the class of interest comprises at least one cancer class. ### Claim 24 (depends on 23) The method of claim 23, wherein the at least one cancer class includes one of the following types of cancer: breast cancer, bladder cancer, brain cancer, lung cancer, pancreatic cancer, skin cancer, colorectal cancer, prostate cancer, stomach cancer, liver cancer, cervical cancer, esophageal cancer, leukemia, non-hodgkin lymphoma, kidney cancer, uterine cancer, bile duct cancer, bone cancer, ovarian cancer, gallbladder cancer, gastrointestinal cancer, oral cancer, throat cancer, ocular cancer, pelvic cancer, spinal cancer, testicular cancer, vaginal cancer, vulvar cancer, or thyroid cancer. ### Claim 25 (depends on 1) The method of claim 1, wherein the class of interest comprises at least one of the following types of tissue: abnormal tissue, benign tissue, malignant tissue, bone tissue, skin tissue, nerve tissue, interstitial tissue, muscle tissue, connective tissue, scar tissue, lymphoid tissue, fat, epithelial tissue, nervous tissue, or blood vessels. ### Claim 26 (depends on 1) The method of claim 1, wherein the digital tissue image comprises a slide image of a tissue sample slide. ### Claim 27 (depends on 26) The method of claim 26, wherein the slide image comprises a digital histopathology image. ### Claim 28 (depends on 26) The method of claim 26, wherein the slide image includes biological sample metadata comprising digital information associated with at least one of the following: a tissue type, a tissue donor, a scanner, a stain, a staining technique, an identifier of a preparer, an image size, a sample identifier, a tracking identifier, a version number, a file type, an image date, a symptom, a diagnosis, an identifying information of treating physician, a medical history of the tissue donor, a demographic information of the tissue donor, a medical history of family of the tissue donor, or a species of the tissue donor. ### Claim 29 An apparatus for generating at least one shape of a region of interest in a digital image, the apparatus comprising: a non-transitory, computer readable memory storing software instructions; and a processor coupled with the computer readable memory, and, upon execution of the software instructions, is configurable to: obtain access to a digital tissue image of a biological sample; tile the digital tissue image into a collection of image patches; obtain a plurality of features from each patch in the collection of image patches, the plurality of features defining a patch feature vector in a multidimensional feature space including the plurality of features as dimensions; determine a user selection of a user selected subset of patches in the collection of image patches, wherein the user selection further includes a selection, prior to selection of the subset of patches in the collection of image patches, of whether the user selected subset of patches are positive or negative patches; classify, by applying a trained classifier to patch vectors of other patches in the collection of patches, the other patches as belonging or not belonging to a same class of interest as the user selected subset of patches; and identify one or more regions of interest based at least in part on the results of the classifying. ### Claim 30 A computer implemented method of generating training data to train a machine learning system, the method comprising: obtaining, by an image processing engine, access to a digital tissue image of a biological sample; tiling, by the image processing engine, the digital tissue image into a collection of image patches; obtaining, by the image processing engine, a plurality of features from each patch in the collection of image patches, the plurality of features defining a patch feature vector in a multidimensional feature space including the plurality of features as dimensions; determining, by the image processing engine, a user selection of a user selected subset of patches in the collection of image patches, wherein the user selection further includes a selection, prior to selection of the subset of patches in the collection of image patches, of whether the user selected subset of patches are positive or negative patches; classifying, by applying a trained generic classifier to patch vectors of other patches in the collection of patches, the other patches as belonging or not belonging to a same class of interest as the user selected subset of patches; and generating, by the image processing engine, training data for the machine learning system based on the classifying of the user-selected and the other patches in the collection of patches.